Overview

Workflow Report

This is a HTML-ification of Workflow state (workflow.state) and the associated files in the workflow output directory. Below is the doc section of the workflow (.workflow file). 

Produce, transform and plot fio-output
======================================

This workflow assumes that the 'prep.workflow' has completed.
The comparison and plots consists of

  * bandwidth in KiB / seconds
  * latency is mean and in nano-seconds
  * IOPS are 'raw' / base-unit

The above in a bar-plot at iodepth=1, and a line-plot as a function of iodepth.

See Artifacts for the generated plots
The workflow started on 26-10-2022, 12:47:42, status on the steps is provided below.

If a workflow step (worklet / run) or test produced any artifacts, then they are listed here.

This is the configuration (.config file) with any variable-placeholders such as {{ local.env.HOME }} filled out. 

devices:
- driver_attachment: kernel
  labels:
  - dev
  - bdev
  - nvm
  - scc
  nsid: 1
  spdk_conf: null
  uri: /dev/nvme0n1
- driver_attachment: kernel
  labels:
  - dev
  - bdev
  - nvm
  - scc
  nsid: 1
  spdk_conf: null
  uri: /dev/nvme1n1
- driver_attachment: kernel
  labels:
  - dev
  - bdev
  - nvm
  - scc
  nsid: 1
  spdk_conf: null
  uri: /dev/nvme2n1
- driver_attachment: kernel
  labels:
  - dev
  - bdev
  - nvm
  - scc
  nsid: 1
  spdk_conf: null
  uri: /dev/nvme3n1
- driver_attachment: kernel
  labels:
  - dev
  - bdev
  - nvm
  - scc
  nsid: 1
  spdk_conf: null
  uri: /dev/nvme4n1
- driver_attachment: kernel
  labels:
  - dev
  - bdev
  - nvm
  - scc
  nsid: 1
  spdk_conf: null
  uri: /dev/nvme5n1
- driver_attachment: kernel
  labels:
  - dev
  - bdev
  - nvm
  - scc
  nsid: 1
  spdk_conf: null
  uri: /dev/nvme6n1
- driver_attachment: kernel
  labels:
  - dev
  - bdev
  - nvm
  - scc
  nsid: 1
  spdk_conf: null
  uri: /dev/nvme7n1
- driver_attachment: kernel
  labels:
  - dev
  - cdev
  - nvm
  - scc
  nsid: 1
  spdk_conf: null
  uri: /dev/ng0n1
- driver_attachment: kernel
  labels:
  - dev
  - cdev
  - nvm
  - scc
  nsid: 1
  spdk_conf: null
  uri: /dev/ng1n1
- driver_attachment: kernel
  labels:
  - dev
  - cdev
  - nvm
  - scc
  nsid: 1
  spdk_conf: null
  uri: /dev/ng2n1
- driver_attachment: kernel
  labels:
  - dev
  - cdev
  - nvm
  - scc
  nsid: 1
  spdk_conf: null
  uri: /dev/ng3n1
- driver_attachment: kernel
  labels:
  - dev
  - cdev
  - nvm
  - scc
  nsid: 1
  spdk_conf: null
  uri: /dev/ng4n1
- driver_attachment: kernel
  labels:
  - dev
  - cdev
  - nvm
  - scc
  nsid: 1
  spdk_conf: null
  uri: /dev/ng5n1
- driver_attachment: kernel
  labels:
  - dev
  - cdev
  - nvm
  - scc
  nsid: 1
  spdk_conf: null
  uri: /dev/ng6n1
- driver_attachment: kernel
  labels:
  - dev
  - cdev
  - nvm
  - scc
  nsid: 1
  spdk_conf: null
  uri: /dev/ng7n1
- driver_attachment: userspace
  labels:
  - dev
  - pcie
  - nvm
  - scc
  nsid: 1
  spdk_conf: null
  uri: '0000:41:00.0'
- driver_attachment: userspace
  labels:
  - dev
  - pcie
  - nvm
  - scc
  nsid: 1
  spdk_conf: null
  uri: '0000:42:00.0'
- driver_attachment: userspace
  labels:
  - dev
  - pcie
  - nvm
  - scc
  nsid: 1
  spdk_conf: null
  uri: '0000:43:00.0'
- driver_attachment: userspace
  labels:
  - dev
  - pcie
  - nvm
  - scc
  nsid: 1
  spdk_conf: null
  uri: '0000:44:00.0'
- driver_attachment: userspace
  labels:
  - dev
  - pcie
  - nvm
  - scc
  nsid: 1
  spdk_conf: null
  uri: '0000:45:00.0'
- driver_attachment: userspace
  labels:
  - dev
  - pcie
  - nvm
  - scc
  nsid: 1
  spdk_conf: null
  uri: '0000:46:00.0'
- driver_attachment: userspace
  labels:
  - dev
  - pcie
  - nvm
  - scc
  nsid: 1
  spdk_conf: null
  uri: '0000:47:00.0'
- driver_attachment: userspace
  labels:
  - dev
  - pcie
  - nvm
  - scc
  nsid: 1
  spdk_conf: null
  uri: '0000:48:00.0'
fio:
  bin: /opt/fio/bin/fio
  build:
    prefix: /opt/fio
  engines:
    io_uring:
      type: builtin
    io_uring_cmd:
      type: builtin
    libaio:
      type: builtin
    spdk_bdev:
      path: /root/git/spdk/build/fio/spdk_bdev
      type: external_preload
    spdk_nvme:
      path: /root/git/spdk/build/fio/spdk_nvme
      type: external_preload
    xnvme:
      path: /usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so
      type: external_dynamic
  repository:
    path: /root/git/fio
    remote: https://github.com/axboe/fio.git
    tag: fio-3.32
liburing:
  repository:
    path: /root/git/liburing
    remote: https://github.com/axboe/liburing.git
    tag: liburing-2.2
os:
  name: debian
  version: bullseye
spdk:
  build:
    prefix: /opt/spdk
  repository:
    branch: for_bdev_xnvme_v6
    path: /root/git/spdk
    remote: https://github.com/safl/spdk.git
transport:
  ssh:
    hostname: fabio
    port: 22
    username: root
xnvme:
  build:
    type: debug
  repository:
    path: /root/git/xnvme
    remote: https://github.com/OpenMPDK/xNVMe.git
    tag: v0.6.0

Steps

Steps uses worklets. A description of the worklet used by the current step is provided below. The description is extracted from the docstring of the worklet and provided here in verbatim / as-is. 


    Run fio...

{'description': '\n    Run fio...\n',
 'extras': {'runlog': {'cmd_01': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_01.output'),
                                  'state': {'begin': 1666781262.0675218,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.8211526870727539,
                                            'end': 1666781262.8886745,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_01.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_01.state')},
                       'cmd_02': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :  '
                                            '150921.39 IOPS      73.69 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :  '
                                            '150921.39 IOPS      73.69 MiB/s\n'
                                            '[2022-10-26 14:45:55.003669] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-26 14:45:55.003716] [ '
                                            'DPDK EAL parameters: [2022-10-26 '
                                            '14:45:55.003722] bdevperf '
                                            '[2022-10-26 14:45:55.003727] '
                                            '--no-shconf [2022-10-26 '
                                            '14:45:55.003731] -l 0 [2022-10-26 '
                                            '14:45:55.003736] --huge-unlink '
                                            '[2022-10-26 14:45:55.003740] '
                                            '--log-level=lib.eal:6 [2022-10-26 '
                                            '14:45:55.003744] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-26 14:45:55.003748] '
                                            '--log-level=user1:6 [2022-10-26 '
                                            '14:45:55.003753] --iova-mode=pa '
                                            '[2022-10-26 14:45:55.003759] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-26 14:45:55.003763] '
                                            '--match-allocations [2022-10-26 '
                                            '14:45:55.003768] '
                                            '--file-prefix=spdk_pid3400 '
                                            '[2022-10-26 14:45:55.003772] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-26 14:45:55.125466] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-26 14:45:55.168002] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-26 14:45:55.168555] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_02.output'),
                                  'state': {'begin': 1666781264.0326226,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
                                                   '-q 1 -o 512 -w randread -t '
                                                   "10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.173058986663818,
                                            'end': 1666781275.2056816,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_02.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_02.state')},
                       'cmd_03': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_03.output'),
                                  'state': {'begin': 1666781275.208141,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.7913839817047119,
                                            'end': 1666781275.999525,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_03.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_03.state')},
                       'cmd_04': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :  '
                                            '171436.90 IOPS      83.71 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :  '
                                            '171436.90 IOPS      83.71 MiB/s\n'
                                            '[2022-10-26 14:46:08.004494] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-26 14:46:08.004535] [ '
                                            'DPDK EAL parameters: [2022-10-26 '
                                            '14:46:08.004541] bdevperf '
                                            '[2022-10-26 14:46:08.004546] '
                                            '--no-shconf [2022-10-26 '
                                            '14:46:08.004550] -l 0 [2022-10-26 '
                                            '14:46:08.004555] --huge-unlink '
                                            '[2022-10-26 14:46:08.004559] '
                                            '--log-level=lib.eal:6 [2022-10-26 '
                                            '14:46:08.004564] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-26 14:46:08.004568] '
                                            '--log-level=user1:6 [2022-10-26 '
                                            '14:46:08.004573] --iova-mode=pa '
                                            '[2022-10-26 14:46:08.004577] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-26 14:46:08.004588] '
                                            '--match-allocations [2022-10-26 '
                                            '14:46:08.004593] '
                                            '--file-prefix=spdk_pid3427 '
                                            '[2022-10-26 14:46:08.004597] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-26 14:46:08.126140] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-26 14:46:08.166933] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-26 14:46:08.167491] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_04.output'),
                                  'state': {'begin': 1666781277.113264,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
                                                   '-q 1 -o 512 -w randread -t '
                                                   "10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.077265501022339,
                                            'end': 1666781288.1905296,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_04.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_04.state')},
                       'cmd_05': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_05.output'),
                                  'state': {'begin': 1666781288.1932762,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.8355164527893066,
                                            'end': 1666781289.0287926,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_05.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_05.state')},
                       'cmd_06': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :  '
                                            '159095.18 IOPS      77.68 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :  '
                                            '159095.18 IOPS      77.68 MiB/s\n'
                                            '[2022-10-26 14:46:21.013312] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-26 14:46:21.013360] [ '
                                            'DPDK EAL parameters: [2022-10-26 '
                                            '14:46:21.013365] bdevperf '
                                            '[2022-10-26 14:46:21.013370] '
                                            '--no-shconf [2022-10-26 '
                                            '14:46:21.013374] -l 0 [2022-10-26 '
                                            '14:46:21.013379] --huge-unlink '
                                            '[2022-10-26 14:46:21.013383] '
                                            '--log-level=lib.eal:6 [2022-10-26 '
                                            '14:46:21.013387] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-26 14:46:21.013391] '
                                            '--log-level=user1:6 [2022-10-26 '
                                            '14:46:21.013396] --iova-mode=pa '
                                            '[2022-10-26 14:46:21.013400] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-26 14:46:21.013404] '
                                            '--match-allocations [2022-10-26 '
                                            '14:46:21.013408] '
                                            '--file-prefix=spdk_pid3453 '
                                            '[2022-10-26 14:46:21.013412] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-26 14:46:21.135461] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-26 14:46:21.187017] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-26 14:46:21.187578] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_06.output'),
                                  'state': {'begin': 1666781290.0950124,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
                                                   '-q 1 -o 512 -w randread -t '
                                                   "10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.119955778121948,
                                            'end': 1666781301.2149682,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_06.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_06.state')},
                       'cmd_07': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_07.output'),
                                  'state': {'begin': 1666781301.2178962,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_uring_foo.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.9140396118164062,
                                            'end': 1666781302.1319358,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_07.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_07.state')},
                       'cmd_08': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :  '
                                            '137262.72 IOPS      67.02 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :  '
                                            '137262.72 IOPS      67.02 MiB/s\n'
                                            '[2022-10-26 14:46:34.208092] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-26 14:46:34.208139] [ '
                                            'DPDK EAL parameters: [2022-10-26 '
                                            '14:46:34.208145] bdevperf '
                                            '[2022-10-26 14:46:34.208150] '
                                            '--no-shconf [2022-10-26 '
                                            '14:46:34.208154] -l 0 [2022-10-26 '
                                            '14:46:34.208158] --huge-unlink '
                                            '[2022-10-26 14:46:34.208164] '
                                            '--log-level=lib.eal:6 [2022-10-26 '
                                            '14:46:34.208168] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-26 14:46:34.208172] '
                                            '--log-level=user1:6 [2022-10-26 '
                                            '14:46:34.208177] --iova-mode=pa '
                                            '[2022-10-26 14:46:34.208181] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-26 14:46:34.208187] '
                                            '--match-allocations [2022-10-26 '
                                            '14:46:34.208191] '
                                            '--file-prefix=spdk_pid3478 '
                                            '[2022-10-26 14:46:34.208195] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-26 14:46:34.330276] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-26 14:46:34.370957] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-26 14:46:34.371536] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_08.output'),
                                  'state': {'begin': 1666781303.3105567,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_uring_foo.conf '
                                                   '-q 1 -o 512 -w randread -t '
                                                   "10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.075981140136719,
                                            'end': 1666781314.3865378,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_08.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_08.state')},
                       'cmd_09': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_09.output'),
                                  'state': {'begin': 1666781314.3892505,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_uring_foo.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.8424861431121826,
                                            'end': 1666781315.2317367,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_09.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_09.state')},
                       'cmd_10': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :  '
                                            '138058.37 IOPS      67.41 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :  '
                                            '138058.37 IOPS      67.41 MiB/s\n'
                                            '[2022-10-26 14:46:47.231816] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-26 14:46:47.231858] [ '
                                            'DPDK EAL parameters: [2022-10-26 '
                                            '14:46:47.231864] bdevperf '
                                            '[2022-10-26 14:46:47.231868] '
                                            '--no-shconf [2022-10-26 '
                                            '14:46:47.231872] -l 0 [2022-10-26 '
                                            '14:46:47.231877] --huge-unlink '
                                            '[2022-10-26 14:46:47.231881] '
                                            '--log-level=lib.eal:6 [2022-10-26 '
                                            '14:46:47.231885] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-26 14:46:47.231889] '
                                            '--log-level=user1:6 [2022-10-26 '
                                            '14:46:47.231893] --iova-mode=pa '
                                            '[2022-10-26 14:46:47.231898] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-26 14:46:47.231902] '
                                            '--match-allocations [2022-10-26 '
                                            '14:46:47.231906] '
                                            '--file-prefix=spdk_pid3503 '
                                            '[2022-10-26 14:46:47.231910] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-26 14:46:47.353497] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-26 14:46:47.394950] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-26 14:46:47.395495] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_10.output'),
                                  'state': {'begin': 1666781316.3184137,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_uring_foo.conf '
                                                   '-q 1 -o 512 -w randread -t '
                                                   "10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.08516263961792,
                                            'end': 1666781327.4035764,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_10.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_10.state')},
                       'cmd_100': {'output': 'Running I/O for 10 seconds...\n'
                                             '\n'
                                             ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                             '\t Nvme0n1             :  '
                                             '482862.92 IOPS     235.77 MiB/s\n'
                                             '\n'
                                             ' '
                                             '=============================================================\n'
                                             '\n'
                                             ' Total                       :  '
                                             '482862.92 IOPS     235.77 MiB/s\n'
                                             '[2022-10-26 14:56:34.681870] '
                                             'Starting SPDK v23.01-pre git '
                                             'sha1 fd41a2937 / DPDK 22.07.0 '
                                             'initialization...\n'
                                             '[2022-10-26 14:56:34.681913] [ '
                                             'DPDK EAL parameters: [2022-10-26 '
                                             '14:56:34.681919] bdevperf '
                                             '[2022-10-26 14:56:34.681924] '
                                             '--no-shconf [2022-10-26 '
                                             '14:56:34.681928] -l 0 '
                                             '[2022-10-26 14:56:34.681932] '
                                             '--huge-unlink [2022-10-26 '
                                             '14:56:34.681937] '
                                             '--log-level=lib.eal:6 '
                                             '[2022-10-26 14:56:34.681941] '
                                             '--log-level=lib.cryptodev:5 '
                                             '[2022-10-26 14:56:34.681945] '
                                             '--log-level=user1:6 [2022-10-26 '
                                             '14:56:34.681949] --iova-mode=pa '
                                             '[2022-10-26 14:56:34.681952] '
                                             '--base-virtaddr=0x200000000000 '
                                             '[2022-10-26 14:56:34.681957] '
                                             '--match-allocations [2022-10-26 '
                                             '14:56:34.681961] '
                                             '--file-prefix=spdk_pid4648 '
                                             '[2022-10-26 14:56:34.681966] ]\n'
                                             'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n'
                                             '[2022-10-26 14:56:34.804031] '
                                             'app.c: 586:spdk_app_start: '
                                             '*NOTICE*: Total cores available: '
                                             '1\n'
                                             '[2022-10-26 14:56:34.842979] '
                                             'reactor.c: 926:reactor_run: '
                                             '*NOTICE*: Reactor started on '
                                             'core 0\n'
                                             '[2022-10-26 14:56:34.843588] '
                                             'accel_sw.c: '
                                             '461:sw_accel_module_init: '
                                             '*NOTICE*: Accel framework '
                                             'software module initialized.\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_100.output'),
                                   'state': {'begin': 1666781903.7373946,
                                             'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                    '--json '
                                                    '/tmp/spdk_bdev_uring_foo.conf '
                                                    '-q 8 -o 512 -w randread '
                                                    "-t 10 -m '[0]'",
                                             'cwd': 'None',
                                             'elapsed': 11.132895708084106,
                                             'end': 1666781914.8702903,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_100.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_100.state')},
                       'cmd_101': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_101.output'),
                                   'state': {'begin': 1666781914.877601,
                                             'cmd': 'rm '
                                                    '/tmp/spdk_bdev_uring_foo.conf '
                                                    '|| true',
                                             'cwd': 'None',
                                             'elapsed': 0.8195874691009521,
                                             'end': 1666781915.6971884,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_101.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_101.state')},
                       'cmd_102': {'output': 'Running I/O for 10 seconds...\n'
                                             '\n'
                                             ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                             '\t Nvme0n1             :  '
                                             '482460.34 IOPS     235.58 MiB/s\n'
                                             '\n'
                                             ' '
                                             '=============================================================\n'
                                             '\n'
                                             ' Total                       :  '
                                             '482460.34 IOPS     235.58 MiB/s\n'
                                             '[2022-10-26 14:56:47.633077] '
                                             'Starting SPDK v23.01-pre git '
                                             'sha1 fd41a2937 / DPDK 22.07.0 '
                                             'initialization...\n'
                                             '[2022-10-26 14:56:47.633125] [ '
                                             'DPDK EAL parameters: [2022-10-26 '
                                             '14:56:47.633131] bdevperf '
                                             '[2022-10-26 14:56:47.633136] '
                                             '--no-shconf [2022-10-26 '
                                             '14:56:47.633141] -l 0 '
                                             '[2022-10-26 14:56:47.633146] '
                                             '--huge-unlink [2022-10-26 '
                                             '14:56:47.633150] '
                                             '--log-level=lib.eal:6 '
                                             '[2022-10-26 14:56:47.633155] '
                                             '--log-level=lib.cryptodev:5 '
                                             '[2022-10-26 14:56:47.633160] '
                                             '--log-level=user1:6 [2022-10-26 '
                                             '14:56:47.633164] --iova-mode=pa '
                                             '[2022-10-26 14:56:47.633169] '
                                             '--base-virtaddr=0x200000000000 '
                                             '[2022-10-26 14:56:47.633174] '
                                             '--match-allocations [2022-10-26 '
                                             '14:56:47.633178] '
                                             '--file-prefix=spdk_pid4673 '
                                             '[2022-10-26 14:56:47.633183] ]\n'
                                             'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n'
                                             '[2022-10-26 14:56:47.755367] '
                                             'app.c: 586:spdk_app_start: '
                                             '*NOTICE*: Total cores available: '
                                             '1\n'
                                             '[2022-10-26 14:56:47.798960] '
                                             'reactor.c: 926:reactor_run: '
                                             '*NOTICE*: Reactor started on '
                                             'core 0\n'
                                             '[2022-10-26 14:56:47.799547] '
                                             'accel_sw.c: '
                                             '461:sw_accel_module_init: '
                                             '*NOTICE*: Accel framework '
                                             'software module initialized.\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_102.output'),
                                   'state': {'begin': 1666781916.7448435,
                                             'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                    '--json '
                                                    '/tmp/spdk_bdev_uring_foo.conf '
                                                    '-q 8 -o 512 -w randread '
                                                    "-t 10 -m '[0]'",
                                             'cwd': 'None',
                                             'elapsed': 11.084971189498901,
                                             'end': 1666781927.8298147,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_102.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_102.state')},
                       'cmd_103': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_103.output'),
                                   'state': {'begin': 1666781927.8339686,
                                             'cmd': 'rm '
                                                    '/tmp/spdk_bdev_xnvme_io_uring.conf '
                                                    '|| true',
                                             'cwd': 'None',
                                             'elapsed': 0.8341481685638428,
                                             'end': 1666781928.6681168,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_103.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_103.state')},
                       'cmd_104': {'output': 'Running I/O for 10 seconds...\n'
                                             '\n'
                                             ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                             '\t Nvme0n1             :  '
                                             '395076.98 IOPS     192.91 MiB/s\n'
                                             '\n'
                                             ' '
                                             '=============================================================\n'
                                             '\n'
                                             ' Total                       :  '
                                             '395076.98 IOPS     192.91 MiB/s\n'
                                             '[2022-10-26 14:57:00.627211] '
                                             'Starting SPDK v23.01-pre git '
                                             'sha1 fd41a2937 / DPDK 22.07.0 '
                                             'initialization...\n'
                                             '[2022-10-26 14:57:00.627256] [ '
                                             'DPDK EAL parameters: [2022-10-26 '
                                             '14:57:00.627262] bdevperf '
                                             '[2022-10-26 14:57:00.627267] '
                                             '--no-shconf [2022-10-26 '
                                             '14:57:00.627271] -l 0 '
                                             '[2022-10-26 14:57:00.627276] '
                                             '--huge-unlink [2022-10-26 '
                                             '14:57:00.627280] '
                                             '--log-level=lib.eal:6 '
                                             '[2022-10-26 14:57:00.627283] '
                                             '--log-level=lib.cryptodev:5 '
                                             '[2022-10-26 14:57:00.627287] '
                                             '--log-level=user1:6 [2022-10-26 '
                                             '14:57:00.627292] --iova-mode=pa '
                                             '[2022-10-26 14:57:00.627297] '
                                             '--base-virtaddr=0x200000000000 '
                                             '[2022-10-26 14:57:00.627301] '
                                             '--match-allocations [2022-10-26 '
                                             '14:57:00.627305] '
                                             '--file-prefix=spdk_pid4698 '
                                             '[2022-10-26 14:57:00.627309] ]\n'
                                             'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n'
                                             '[2022-10-26 14:57:00.749257] '
                                             'app.c: 586:spdk_app_start: '
                                             '*NOTICE*: Total cores available: '
                                             '1\n'
                                             '[2022-10-26 14:57:00.791057] '
                                             'reactor.c: 926:reactor_run: '
                                             '*NOTICE*: Reactor started on '
                                             'core 0\n'
                                             '[2022-10-26 14:57:00.791650] '
                                             'accel_sw.c: '
                                             '461:sw_accel_module_init: '
                                             '*NOTICE*: Accel framework '
                                             'software module initialized.\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_104.output'),
                                   'state': {'begin': 1666781929.7034733,
                                             'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                    '--json '
                                                    '/tmp/spdk_bdev_xnvme_io_uring.conf '
                                                    '-q 8 -o 512 -w randread '
                                                    "-t 10 -m '[0]'",
                                             'cwd': 'None',
                                             'elapsed': 11.11667513847351,
                                             'end': 1666781940.8201485,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_104.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_104.state')},
                       'cmd_105': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_105.output'),
                                   'state': {'begin': 1666781940.8240077,
                                             'cmd': 'rm '
                                                    '/tmp/spdk_bdev_xnvme_io_uring.conf '
                                                    '|| true',
                                             'cwd': 'None',
                                             'elapsed': 0.8417377471923828,
                                             'end': 1666781941.6657455,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_105.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_105.state')},
                       'cmd_106': {'output': 'Running I/O for 10 seconds...\n'
                                             '\n'
                                             ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                             '\t Nvme0n1             :  '
                                             '392195.29 IOPS     191.50 MiB/s\n'
                                             '\n'
                                             ' '
                                             '=============================================================\n'
                                             '\n'
                                             ' Total                       :  '
                                             '392195.29 IOPS     191.50 MiB/s\n'
                                             '[2022-10-26 14:57:13.666086] '
                                             'Starting SPDK v23.01-pre git '
                                             'sha1 fd41a2937 / DPDK 22.07.0 '
                                             'initialization...\n'
                                             '[2022-10-26 14:57:13.666140] [ '
                                             'DPDK EAL parameters: [2022-10-26 '
                                             '14:57:13.666148] bdevperf '
                                             '[2022-10-26 14:57:13.666155] '
                                             '--no-shconf [2022-10-26 '
                                             '14:57:13.666161] -l 0 '
                                             '[2022-10-26 14:57:13.666168] '
                                             '--huge-unlink [2022-10-26 '
                                             '14:57:13.666173] '
                                             '--log-level=lib.eal:6 '
                                             '[2022-10-26 14:57:13.666179] '
                                             '--log-level=lib.cryptodev:5 '
                                             '[2022-10-26 14:57:13.666185] '
                                             '--log-level=user1:6 [2022-10-26 '
                                             '14:57:13.666192] --iova-mode=pa '
                                             '[2022-10-26 14:57:13.666199] '
                                             '--base-virtaddr=0x200000000000 '
                                             '[2022-10-26 14:57:13.666206] '
                                             '--match-allocations [2022-10-26 '
                                             '14:57:13.666213] '
                                             '--file-prefix=spdk_pid4723 '
                                             '[2022-10-26 14:57:13.666219] ]\n'
                                             'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n'
                                             '[2022-10-26 14:57:13.788213] '
                                             'app.c: 586:spdk_app_start: '
                                             '*NOTICE*: Total cores available: '
                                             '1\n'
                                             '[2022-10-26 14:57:13.831019] '
                                             'reactor.c: 926:reactor_run: '
                                             '*NOTICE*: Reactor started on '
                                             'core 0\n'
                                             '[2022-10-26 14:57:13.831606] '
                                             'accel_sw.c: '
                                             '461:sw_accel_module_init: '
                                             '*NOTICE*: Accel framework '
                                             'software module initialized.\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_106.output'),
                                   'state': {'begin': 1666781942.728097,
                                             'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                    '--json '
                                                    '/tmp/spdk_bdev_xnvme_io_uring.conf '
                                                    '-q 8 -o 512 -w randread '
                                                    "-t 10 -m '[0]'",
                                             'cwd': 'None',
                                             'elapsed': 11.124542474746704,
                                             'end': 1666781953.8526394,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_106.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_106.state')},
                       'cmd_107': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_107.output'),
                                   'state': {'begin': 1666781953.8553784,
                                             'cmd': 'rm '
                                                    '/tmp/spdk_bdev_xnvme_io_uring.conf '
                                                    '|| true',
                                             'cwd': 'None',
                                             'elapsed': 0.7792153358459473,
                                             'end': 1666781954.6345937,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_107.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_107.state')},
                       'cmd_108': {'output': 'Running I/O for 10 seconds...\n'
                                             '\n'
                                             ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                             '\t Nvme0n1             :  '
                                             '390867.35 IOPS     190.85 MiB/s\n'
                                             '\n'
                                             ' '
                                             '=============================================================\n'
                                             '\n'
                                             ' Total                       :  '
                                             '390867.35 IOPS     190.85 MiB/s\n'
                                             '[2022-10-26 14:57:26.619274] '
                                             'Starting SPDK v23.01-pre git '
                                             'sha1 fd41a2937 / DPDK 22.07.0 '
                                             'initialization...\n'
                                             '[2022-10-26 14:57:26.619322] [ '
                                             'DPDK EAL parameters: [2022-10-26 '
                                             '14:57:26.619328] bdevperf '
                                             '[2022-10-26 14:57:26.619332] '
                                             '--no-shconf [2022-10-26 '
                                             '14:57:26.619337] -l 0 '
                                             '[2022-10-26 14:57:26.619342] '
                                             '--huge-unlink [2022-10-26 '
                                             '14:57:26.619348] '
                                             '--log-level=lib.eal:6 '
                                             '[2022-10-26 14:57:26.619353] '
                                             '--log-level=lib.cryptodev:5 '
                                             '[2022-10-26 14:57:26.619358] '
                                             '--log-level=user1:6 [2022-10-26 '
                                             '14:57:26.619362] --iova-mode=pa '
                                             '[2022-10-26 14:57:26.619366] '
                                             '--base-virtaddr=0x200000000000 '
                                             '[2022-10-26 14:57:26.619371] '
                                             '--match-allocations [2022-10-26 '
                                             '14:57:26.619376] '
                                             '--file-prefix=spdk_pid4748 '
                                             '[2022-10-26 14:57:26.619380] ]\n'
                                             'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n'
                                             '[2022-10-26 14:57:26.741441] '
                                             'app.c: 586:spdk_app_start: '
                                             '*NOTICE*: Total cores available: '
                                             '1\n'
                                             '[2022-10-26 14:57:26.791036] '
                                             'reactor.c: 926:reactor_run: '
                                             '*NOTICE*: Reactor started on '
                                             'core 0\n'
                                             '[2022-10-26 14:57:26.791643] '
                                             'accel_sw.c: '
                                             '461:sw_accel_module_init: '
                                             '*NOTICE*: Accel framework '
                                             'software module initialized.\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_108.output'),
                                   'state': {'begin': 1666781955.6668162,
                                             'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                    '--json '
                                                    '/tmp/spdk_bdev_xnvme_io_uring.conf '
                                                    '-q 8 -o 512 -w randread '
                                                    "-t 10 -m '[0]'",
                                             'cwd': 'None',
                                             'elapsed': 11.153231143951416,
                                             'end': 1666781966.8200474,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_108.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_108.state')},
                       'cmd_109': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_109.output'),
                                   'state': {'begin': 1666781966.8233027,
                                             'cmd': 'rm '
                                                    '/tmp/spdk_bdev_aio_foo.conf '
                                                    '|| true',
                                             'cwd': 'None',
                                             'elapsed': 0.7791199684143066,
                                             'end': 1666781967.6024227,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_109.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_109.state')},
                       'cmd_11': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_11.output'),
                                  'state': {'begin': 1666781327.4065323,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_uring_foo.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.8043515682220459,
                                            'end': 1666781328.2108839,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_11.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_11.state')},
                       'cmd_110': {'output': 'Running I/O for 10 seconds...\n'
                                             '\n'
                                             ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                             '\t Nvme0n1             :  '
                                             '404566.90 IOPS     197.54 MiB/s\n'
                                             '\n'
                                             ' '
                                             '=============================================================\n'
                                             '\n'
                                             ' Total                       :  '
                                             '404566.90 IOPS     197.54 MiB/s\n'
                                             '[2022-10-26 14:57:39.566766] '
                                             'Starting SPDK v23.01-pre git '
                                             'sha1 fd41a2937 / DPDK 22.07.0 '
                                             'initialization...\n'
                                             '[2022-10-26 14:57:39.566809] [ '
                                             'DPDK EAL parameters: [2022-10-26 '
                                             '14:57:39.566815] bdevperf '
                                             '[2022-10-26 14:57:39.566821] '
                                             '--no-shconf [2022-10-26 '
                                             '14:57:39.566825] -l 0 '
                                             '[2022-10-26 14:57:39.566829] '
                                             '--huge-unlink [2022-10-26 '
                                             '14:57:39.566834] '
                                             '--log-level=lib.eal:6 '
                                             '[2022-10-26 14:57:39.566838] '
                                             '--log-level=lib.cryptodev:5 '
                                             '[2022-10-26 14:57:39.566842] '
                                             '--log-level=user1:6 [2022-10-26 '
                                             '14:57:39.566846] --iova-mode=pa '
                                             '[2022-10-26 14:57:39.566850] '
                                             '--base-virtaddr=0x200000000000 '
                                             '[2022-10-26 14:57:39.566855] '
                                             '--match-allocations [2022-10-26 '
                                             '14:57:39.566859] '
                                             '--file-prefix=spdk_pid4773 '
                                             '[2022-10-26 14:57:39.566863] ]\n'
                                             'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n'
                                             '[2022-10-26 14:57:39.688845] '
                                             'app.c: 586:spdk_app_start: '
                                             '*NOTICE*: Total cores available: '
                                             '1\n'
                                             '[2022-10-26 14:57:39.731022] '
                                             'reactor.c: 926:reactor_run: '
                                             '*NOTICE*: Reactor started on '
                                             'core 0\n'
                                             '[2022-10-26 14:57:39.731598] '
                                             'accel_sw.c: '
                                             '461:sw_accel_module_init: '
                                             '*NOTICE*: Accel framework '
                                             'software module initialized.\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_110.output'),
                                   'state': {'begin': 1666781968.6418266,
                                             'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                    '--json '
                                                    '/tmp/spdk_bdev_aio_foo.conf '
                                                    '-q 8 -o 512 -w randread '
                                                    "-t 10 -m '[0]'",
                                             'cwd': 'None',
                                             'elapsed': 11.212659120559692,
                                             'end': 1666781979.8544858,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_110.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_110.state')},
                       'cmd_111': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_111.output'),
                                   'state': {'begin': 1666781979.857094,
                                             'cmd': 'rm '
                                                    '/tmp/spdk_bdev_aio_foo.conf '
                                                    '|| true',
                                             'cwd': 'None',
                                             'elapsed': 0.7954108715057373,
                                             'end': 1666781980.652505,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_111.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_111.state')},
                       'cmd_112': {'output': 'Running I/O for 10 seconds...\n'
                                             '\n'
                                             ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                             '\t Nvme0n1             :  '
                                             '404402.85 IOPS     197.46 MiB/s\n'
                                             '\n'
                                             ' '
                                             '=============================================================\n'
                                             '\n'
                                             ' Total                       :  '
                                             '404402.85 IOPS     197.46 MiB/s\n'
                                             '[2022-10-26 14:57:52.662483] '
                                             'Starting SPDK v23.01-pre git '
                                             'sha1 fd41a2937 / DPDK 22.07.0 '
                                             'initialization...\n'
                                             '[2022-10-26 14:57:52.662525] [ '
                                             'DPDK EAL parameters: [2022-10-26 '
                                             '14:57:52.662531] bdevperf '
                                             '[2022-10-26 14:57:52.662536] '
                                             '--no-shconf [2022-10-26 '
                                             '14:57:52.662540] -l 0 '
                                             '[2022-10-26 14:57:52.662545] '
                                             '--huge-unlink [2022-10-26 '
                                             '14:57:52.662549] '
                                             '--log-level=lib.eal:6 '
                                             '[2022-10-26 14:57:52.662553] '
                                             '--log-level=lib.cryptodev:5 '
                                             '[2022-10-26 14:57:52.662557] '
                                             '--log-level=user1:6 [2022-10-26 '
                                             '14:57:52.662561] --iova-mode=pa '
                                             '[2022-10-26 14:57:52.662565] '
                                             '--base-virtaddr=0x200000000000 '
                                             '[2022-10-26 14:57:52.662570] '
                                             '--match-allocations [2022-10-26 '
                                             '14:57:52.662574] '
                                             '--file-prefix=spdk_pid4799 '
                                             '[2022-10-26 14:57:52.662579] ]\n'
                                             'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n'
                                             '[2022-10-26 14:57:52.784566] '
                                             'app.c: 586:spdk_app_start: '
                                             '*NOTICE*: Total cores available: '
                                             '1\n'
                                             '[2022-10-26 14:57:52.827141] '
                                             'reactor.c: 926:reactor_run: '
                                             '*NOTICE*: Reactor started on '
                                             'core 0\n'
                                             '[2022-10-26 14:57:52.827734] '
                                             'accel_sw.c: '
                                             '461:sw_accel_module_init: '
                                             '*NOTICE*: Accel framework '
                                             'software module initialized.\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_112.output'),
                                   'state': {'begin': 1666781981.7609088,
                                             'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                    '--json '
                                                    '/tmp/spdk_bdev_aio_foo.conf '
                                                    '-q 8 -o 512 -w randread '
                                                    "-t 10 -m '[0]'",
                                             'cwd': 'None',
                                             'elapsed': 11.155568361282349,
                                             'end': 1666781992.9164772,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_112.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_112.state')},
                       'cmd_113': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_113.output'),
                                   'state': {'begin': 1666781992.9190931,
                                             'cmd': 'rm '
                                                    '/tmp/spdk_bdev_aio_foo.conf '
                                                    '|| true',
                                             'cwd': 'None',
                                             'elapsed': 0.8321177959442139,
                                             'end': 1666781993.751211,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_113.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_113.state')},
                       'cmd_114': {'output': 'Running I/O for 10 seconds...\n'
                                             '\n'
                                             ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                             '\t Nvme0n1             :  '
                                             '403868.55 IOPS     197.20 MiB/s\n'
                                             '\n'
                                             ' '
                                             '=============================================================\n'
                                             '\n'
                                             ' Total                       :  '
                                             '403868.55 IOPS     197.20 MiB/s\n'
                                             '[2022-10-26 14:58:05.759802] '
                                             'Starting SPDK v23.01-pre git '
                                             'sha1 fd41a2937 / DPDK 22.07.0 '
                                             'initialization...\n'
                                             '[2022-10-26 14:58:05.759849] [ '
                                             'DPDK EAL parameters: [2022-10-26 '
                                             '14:58:05.759855] bdevperf '
                                             '[2022-10-26 14:58:05.759860] '
                                             '--no-shconf [2022-10-26 '
                                             '14:58:05.759864] -l 0 '
                                             '[2022-10-26 14:58:05.759869] '
                                             '--huge-unlink [2022-10-26 '
                                             '14:58:05.759874] '
                                             '--log-level=lib.eal:6 '
                                             '[2022-10-26 14:58:05.759878] '
                                             '--log-level=lib.cryptodev:5 '
                                             '[2022-10-26 14:58:05.759883] '
                                             '--log-level=user1:6 [2022-10-26 '
                                             '14:58:05.759887] --iova-mode=pa '
                                             '[2022-10-26 14:58:05.759891] '
                                             '--base-virtaddr=0x200000000000 '
                                             '[2022-10-26 14:58:05.759895] '
                                             '--match-allocations [2022-10-26 '
                                             '14:58:05.759900] '
                                             '--file-prefix=spdk_pid4825 '
                                             '[2022-10-26 14:58:05.759904] ]\n'
                                             'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n'
                                             '[2022-10-26 14:58:05.881869] '
                                             'app.c: 586:spdk_app_start: '
                                             '*NOTICE*: Total cores available: '
                                             '1\n'
                                             '[2022-10-26 14:58:05.919038] '
                                             'reactor.c: 926:reactor_run: '
                                             '*NOTICE*: Reactor started on '
                                             'core 0\n'
                                             '[2022-10-26 14:58:05.919619] '
                                             'accel_sw.c: '
                                             '461:sw_accel_module_init: '
                                             '*NOTICE*: Accel framework '
                                             'software module initialized.\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_114.output'),
                                   'state': {'begin': 1666781994.8151395,
                                             'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                    '--json '
                                                    '/tmp/spdk_bdev_aio_foo.conf '
                                                    '-q 8 -o 512 -w randread '
                                                    "-t 10 -m '[0]'",
                                             'cwd': 'None',
                                             'elapsed': 11.180953979492188,
                                             'end': 1666782005.9960935,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_114.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_114.state')},
                       'cmd_115': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_115.output'),
                                   'state': {'begin': 1666782005.9991379,
                                             'cmd': 'rm '
                                                    '/tmp/spdk_bdev_xnvme_libaio.conf '
                                                    '|| true',
                                             'cwd': 'None',
                                             'elapsed': 0.818833589553833,
                                             'end': 1666782006.8179715,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_115.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_115.state')},
                       'cmd_116': {'output': 'Running I/O for 10 seconds...\n'
                                             '\n'
                                             ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                             '\t Nvme0n1             :  '
                                             '382108.42 IOPS     186.58 MiB/s\n'
                                             '\n'
                                             ' '
                                             '=============================================================\n'
                                             '\n'
                                             ' Total                       :  '
                                             '382108.42 IOPS     186.58 MiB/s\n'
                                             '[2022-10-26 14:58:18.828990] '
                                             'Starting SPDK v23.01-pre git '
                                             'sha1 fd41a2937 / DPDK 22.07.0 '
                                             'initialization...\n'
                                             '[2022-10-26 14:58:18.829036] [ '
                                             'DPDK EAL parameters: [2022-10-26 '
                                             '14:58:18.829042] bdevperf '
                                             '[2022-10-26 14:58:18.829047] '
                                             '--no-shconf [2022-10-26 '
                                             '14:58:18.829051] -l 0 '
                                             '[2022-10-26 14:58:18.829056] '
                                             '--huge-unlink [2022-10-26 '
                                             '14:58:18.829061] '
                                             '--log-level=lib.eal:6 '
                                             '[2022-10-26 14:58:18.829065] '
                                             '--log-level=lib.cryptodev:5 '
                                             '[2022-10-26 14:58:18.829070] '
                                             '--log-level=user1:6 [2022-10-26 '
                                             '14:58:18.829074] --iova-mode=pa '
                                             '[2022-10-26 14:58:18.829078] '
                                             '--base-virtaddr=0x200000000000 '
                                             '[2022-10-26 14:58:18.829083] '
                                             '--match-allocations [2022-10-26 '
                                             '14:58:18.829088] '
                                             '--file-prefix=spdk_pid4850 '
                                             '[2022-10-26 14:58:18.829092] ]\n'
                                             'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n'
                                             '[2022-10-26 14:58:18.951199] '
                                             'app.c: 586:spdk_app_start: '
                                             '*NOTICE*: Total cores available: '
                                             '1\n'
                                             '[2022-10-26 14:58:18.995101] '
                                             'reactor.c: 926:reactor_run: '
                                             '*NOTICE*: Reactor started on '
                                             'core 0\n'
                                             '[2022-10-26 14:58:18.995684] '
                                             'accel_sw.c: '
                                             '461:sw_accel_module_init: '
                                             '*NOTICE*: Accel framework '
                                             'software module initialized.\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_116.output'),
                                   'state': {'begin': 1666782007.8945894,
                                             'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                    '--json '
                                                    '/tmp/spdk_bdev_xnvme_libaio.conf '
                                                    '-q 8 -o 512 -w randread '
                                                    "-t 10 -m '[0]'",
                                             'cwd': 'None',
                                             'elapsed': 11.20103645324707,
                                             'end': 1666782019.0956259,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_116.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_116.state')},
                       'cmd_117': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_117.output'),
                                   'state': {'begin': 1666782019.098346,
                                             'cmd': 'rm '
                                                    '/tmp/spdk_bdev_xnvme_libaio.conf '
                                                    '|| true',
                                             'cwd': 'None',
                                             'elapsed': 1.1820051670074463,
                                             'end': 1666782020.2803512,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_117.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_117.state')},
                       'cmd_118': {'output': 'Running I/O for 10 seconds...\n'
                                             '\n'
                                             ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                             '\t Nvme0n1             :  '
                                             '380577.08 IOPS     185.83 MiB/s\n'
                                             '\n'
                                             ' '
                                             '=============================================================\n'
                                             '\n'
                                             ' Total                       :  '
                                             '380577.08 IOPS     185.83 MiB/s\n'
                                             '[2022-10-26 14:58:32.287850] '
                                             'Starting SPDK v23.01-pre git '
                                             'sha1 fd41a2937 / DPDK 22.07.0 '
                                             'initialization...\n'
                                             '[2022-10-26 14:58:32.287904] [ '
                                             'DPDK EAL parameters: [2022-10-26 '
                                             '14:58:32.287910] bdevperf '
                                             '[2022-10-26 14:58:32.287915] '
                                             '--no-shconf [2022-10-26 '
                                             '14:58:32.287920] -l 0 '
                                             '[2022-10-26 14:58:32.287924] '
                                             '--huge-unlink [2022-10-26 '
                                             '14:58:32.287929] '
                                             '--log-level=lib.eal:6 '
                                             '[2022-10-26 14:58:32.287933] '
                                             '--log-level=lib.cryptodev:5 '
                                             '[2022-10-26 14:58:32.287937] '
                                             '--log-level=user1:6 [2022-10-26 '
                                             '14:58:32.287942] --iova-mode=pa '
                                             '[2022-10-26 14:58:32.287946] '
                                             '--base-virtaddr=0x200000000000 '
                                             '[2022-10-26 14:58:32.287950] '
                                             '--match-allocations [2022-10-26 '
                                             '14:58:32.287955] '
                                             '--file-prefix=spdk_pid4875 '
                                             '[2022-10-26 14:58:32.287960] ]\n'
                                             'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n'
                                             '[2022-10-26 14:58:32.409866] '
                                             'app.c: 586:spdk_app_start: '
                                             '*NOTICE*: Total cores available: '
                                             '1\n'
                                             '[2022-10-26 14:58:32.450980] '
                                             'reactor.c: 926:reactor_run: '
                                             '*NOTICE*: Reactor started on '
                                             'core 0\n'
                                             '[2022-10-26 14:58:32.451586] '
                                             'accel_sw.c: '
                                             '461:sw_accel_module_init: '
                                             '*NOTICE*: Accel framework '
                                             'software module initialized.\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_118.output'),
                                   'state': {'begin': 1666782021.3781052,
                                             'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                    '--json '
                                                    '/tmp/spdk_bdev_xnvme_libaio.conf '
                                                    '-q 8 -o 512 -w randread '
                                                    "-t 10 -m '[0]'",
                                             'cwd': 'None',
                                             'elapsed': 11.193742275238037,
                                             'end': 1666782032.5718474,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_118.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_118.state')},
                       'cmd_119': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_119.output'),
                                   'state': {'begin': 1666782032.5745347,
                                             'cmd': 'rm '
                                                    '/tmp/spdk_bdev_xnvme_libaio.conf '
                                                    '|| true',
                                             'cwd': 'None',
                                             'elapsed': 0.8060545921325684,
                                             'end': 1666782033.3805892,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_119.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_119.state')},
                       'cmd_12': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :  '
                                            '137312.00 IOPS      67.05 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :  '
                                            '137312.00 IOPS      67.05 MiB/s\n'
                                            '[2022-10-26 14:47:00.239018] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-26 14:47:00.239062] [ '
                                            'DPDK EAL parameters: [2022-10-26 '
                                            '14:47:00.239067] bdevperf '
                                            '[2022-10-26 14:47:00.239073] '
                                            '--no-shconf [2022-10-26 '
                                            '14:47:00.239077] -l 0 [2022-10-26 '
                                            '14:47:00.239082] --huge-unlink '
                                            '[2022-10-26 14:47:00.239086] '
                                            '--log-level=lib.eal:6 [2022-10-26 '
                                            '14:47:00.239091] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-26 14:47:00.239096] '
                                            '--log-level=user1:6 [2022-10-26 '
                                            '14:47:00.239100] --iova-mode=pa '
                                            '[2022-10-26 14:47:00.239104] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-26 14:47:00.239108] '
                                            '--match-allocations [2022-10-26 '
                                            '14:47:00.239112] '
                                            '--file-prefix=spdk_pid3528 '
                                            '[2022-10-26 14:47:00.239117] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-26 14:47:00.360863] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-26 14:47:00.406981] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-26 14:47:00.407550] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_12.output'),
                                  'state': {'begin': 1666781329.3192575,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_uring_foo.conf '
                                                   '-q 1 -o 512 -w randread -t '
                                                   "10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.100764513015747,
                                            'end': 1666781340.420022,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_12.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_12.state')},
                       'cmd_120': {'output': 'Running I/O for 10 seconds...\n'
                                             '\n'
                                             ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                             '\t Nvme0n1             :  '
                                             '379284.08 IOPS     185.20 MiB/s\n'
                                             '\n'
                                             ' '
                                             '=============================================================\n'
                                             '\n'
                                             ' Total                       :  '
                                             '379284.08 IOPS     185.20 MiB/s\n'
                                             '[2022-10-26 14:58:45.398827] '
                                             'Starting SPDK v23.01-pre git '
                                             'sha1 fd41a2937 / DPDK 22.07.0 '
                                             'initialization...\n'
                                             '[2022-10-26 14:58:45.398870] [ '
                                             'DPDK EAL parameters: [2022-10-26 '
                                             '14:58:45.398876] bdevperf '
                                             '[2022-10-26 14:58:45.398881] '
                                             '--no-shconf [2022-10-26 '
                                             '14:58:45.398885] -l 0 '
                                             '[2022-10-26 14:58:45.398890] '
                                             '--huge-unlink [2022-10-26 '
                                             '14:58:45.398895] '
                                             '--log-level=lib.eal:6 '
                                             '[2022-10-26 14:58:45.398899] '
                                             '--log-level=lib.cryptodev:5 '
                                             '[2022-10-26 14:58:45.398904] '
                                             '--log-level=user1:6 [2022-10-26 '
                                             '14:58:45.398908] --iova-mode=pa '
                                             '[2022-10-26 14:58:45.398912] '
                                             '--base-virtaddr=0x200000000000 '
                                             '[2022-10-26 14:58:45.398918] '
                                             '--match-allocations [2022-10-26 '
                                             '14:58:45.398924] '
                                             '--file-prefix=spdk_pid4900 '
                                             '[2022-10-26 14:58:45.398929] ]\n'
                                             'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n'
                                             '[2022-10-26 14:58:45.521201] '
                                             'app.c: 586:spdk_app_start: '
                                             '*NOTICE*: Total cores available: '
                                             '1\n'
                                             '[2022-10-26 14:58:45.571063] '
                                             'reactor.c: 926:reactor_run: '
                                             '*NOTICE*: Reactor started on '
                                             'core 0\n'
                                             '[2022-10-26 14:58:45.571640] '
                                             'accel_sw.c: '
                                             '461:sw_accel_module_init: '
                                             '*NOTICE*: Accel framework '
                                             'software module initialized.\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_120.output'),
                                   'state': {'begin': 1666782034.445311,
                                             'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                    '--json '
                                                    '/tmp/spdk_bdev_xnvme_libaio.conf '
                                                    '-q 8 -o 512 -w randread '
                                                    "-t 10 -m '[0]'",
                                             'cwd': 'None',
                                             'elapsed': 11.227607488632202,
                                             'end': 1666782045.6729186,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_120.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_120.state')},
                       'cmd_13': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_13.output'),
                                  'state': {'begin': 1666781340.4228358,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_xnvme_io_uring.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.7960231304168701,
                                            'end': 1666781341.218859,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_13.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_13.state')},
                       'cmd_14': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :  '
                                            '156589.41 IOPS      76.46 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :  '
                                            '156589.41 IOPS      76.46 MiB/s\n'
                                            '[2022-10-26 14:47:13.264251] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-26 14:47:13.264293] [ '
                                            'DPDK EAL parameters: [2022-10-26 '
                                            '14:47:13.264298] bdevperf '
                                            '[2022-10-26 14:47:13.264303] '
                                            '--no-shconf [2022-10-26 '
                                            '14:47:13.264307] -l 0 [2022-10-26 '
                                            '14:47:13.264311] --huge-unlink '
                                            '[2022-10-26 14:47:13.264316] '
                                            '--log-level=lib.eal:6 [2022-10-26 '
                                            '14:47:13.264320] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-26 14:47:13.264324] '
                                            '--log-level=user1:6 [2022-10-26 '
                                            '14:47:13.264328] --iova-mode=pa '
                                            '[2022-10-26 14:47:13.264332] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-26 14:47:13.264336] '
                                            '--match-allocations [2022-10-26 '
                                            '14:47:13.264340] '
                                            '--file-prefix=spdk_pid3553 '
                                            '[2022-10-26 14:47:13.264344] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-26 14:47:13.386330] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-26 14:47:13.430961] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-26 14:47:13.431535] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_14.output'),
                                  'state': {'begin': 1666781342.3381026,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_xnvme_io_uring.conf '
                                                   '-q 1 -o 512 -w randread -t '
                                                   "10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.116260051727295,
                                            'end': 1666781353.4543626,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_14.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_14.state')},
                       'cmd_15': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_15.output'),
                                  'state': {'begin': 1666781353.4572272,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_xnvme_io_uring.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.8200883865356445,
                                            'end': 1666781354.2773156,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_15.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_15.state')},
                       'cmd_16': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :  '
                                            '156437.11 IOPS      76.39 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :  '
                                            '156437.11 IOPS      76.39 MiB/s\n'
                                            '[2022-10-26 14:47:26.307921] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-26 14:47:26.307971] [ '
                                            'DPDK EAL parameters: [2022-10-26 '
                                            '14:47:26.307978] bdevperf '
                                            '[2022-10-26 14:47:26.307985] '
                                            '--no-shconf [2022-10-26 '
                                            '14:47:26.307992] -l 0 [2022-10-26 '
                                            '14:47:26.307997] --huge-unlink '
                                            '[2022-10-26 14:47:26.308003] '
                                            '--log-level=lib.eal:6 [2022-10-26 '
                                            '14:47:26.308009] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-26 14:47:26.308016] '
                                            '--log-level=user1:6 [2022-10-26 '
                                            '14:47:26.308022] --iova-mode=pa '
                                            '[2022-10-26 14:47:26.308028] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-26 14:47:26.308034] '
                                            '--match-allocations [2022-10-26 '
                                            '14:47:26.308041] '
                                            '--file-prefix=spdk_pid3578 '
                                            '[2022-10-26 14:47:26.308053] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-26 14:47:26.429664] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-26 14:47:26.462979] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-26 14:47:26.463535] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_16.output'),
                                  'state': {'begin': 1666781355.4097142,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_xnvme_io_uring.conf '
                                                   '-q 1 -o 512 -w randread -t '
                                                   "10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.081530332565308,
                                            'end': 1666781366.4912446,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_16.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_16.state')},
                       'cmd_17': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_17.output'),
                                  'state': {'begin': 1666781366.4939313,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_xnvme_io_uring.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.7980461120605469,
                                            'end': 1666781367.2919774,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_17.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_17.state')},
                       'cmd_18': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :  '
                                            '156357.41 IOPS      76.35 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :  '
                                            '156357.41 IOPS      76.35 MiB/s\n'
                                            '[2022-10-26 14:47:39.245614] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-26 14:47:39.245656] [ '
                                            'DPDK EAL parameters: [2022-10-26 '
                                            '14:47:39.245662] bdevperf '
                                            '[2022-10-26 14:47:39.245667] '
                                            '--no-shconf [2022-10-26 '
                                            '14:47:39.245672] -l 0 [2022-10-26 '
                                            '14:47:39.245676] --huge-unlink '
                                            '[2022-10-26 14:47:39.245681] '
                                            '--log-level=lib.eal:6 [2022-10-26 '
                                            '14:47:39.245685] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-26 14:47:39.245689] '
                                            '--log-level=user1:6 [2022-10-26 '
                                            '14:47:39.245694] --iova-mode=pa '
                                            '[2022-10-26 14:47:39.245698] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-26 14:47:39.245703] '
                                            '--match-allocations [2022-10-26 '
                                            '14:47:39.245708] '
                                            '--file-prefix=spdk_pid3603 '
                                            '[2022-10-26 14:47:39.245712] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-26 14:47:39.367532] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-26 14:47:39.411007] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-26 14:47:39.411561] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_18.output'),
                                  'state': {'begin': 1666781368.3570135,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_xnvme_io_uring.conf '
                                                   '-q 1 -o 512 -w randread -t '
                                                   "10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.065926790237427,
                                            'end': 1666781379.4229403,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_18.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_18.state')},
                       'cmd_19': {'output': 'rm: cannot remove '
                                            "'/tmp/spdk_bdev_aio_foo.conf': No "
                                            'such file or directory\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_19.output'),
                                  'state': {'begin': 1666781379.4257424,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_aio_foo.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.8077173233032227,
                                            'end': 1666781380.2334597,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_19.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_19.state')},
                       'cmd_20': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :  '
                                            '154952.02 IOPS      75.66 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :  '
                                            '154952.02 IOPS      75.66 MiB/s\n'
                                            '[2022-10-26 14:47:52.225176] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-26 14:47:52.225221] [ '
                                            'DPDK EAL parameters: [2022-10-26 '
                                            '14:47:52.225227] bdevperf '
                                            '[2022-10-26 14:47:52.225232] '
                                            '--no-shconf [2022-10-26 '
                                            '14:47:52.225236] -l 0 [2022-10-26 '
                                            '14:47:52.225240] --huge-unlink '
                                            '[2022-10-26 14:47:52.225245] '
                                            '--log-level=lib.eal:6 [2022-10-26 '
                                            '14:47:52.225249] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-26 14:47:52.225253] '
                                            '--log-level=user1:6 [2022-10-26 '
                                            '14:47:52.225257] --iova-mode=pa '
                                            '[2022-10-26 14:47:52.225261] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-26 14:47:52.225265] '
                                            '--match-allocations [2022-10-26 '
                                            '14:47:52.225270] '
                                            '--file-prefix=spdk_pid3628 '
                                            '[2022-10-26 14:47:52.225276] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-26 14:47:52.346983] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-26 14:47:52.398946] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-26 14:47:52.399501] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_20.output'),
                                  'state': {'begin': 1666781381.2815309,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_aio_foo.conf '
                                                   '-q 1 -o 512 -w randread -t '
                                                   "10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.207417964935303,
                                            'end': 1666781392.4889488,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_20.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_20.state')},
                       'cmd_21': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_21.output'),
                                  'state': {'begin': 1666781392.4910908,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_aio_foo.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.7867584228515625,
                                            'end': 1666781393.2778492,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_21.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_21.state')},
                       'cmd_22': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :  '
                                            '154636.84 IOPS      75.51 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :  '
                                            '154636.84 IOPS      75.51 MiB/s\n'
                                            '[2022-10-26 14:48:05.239169] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-26 14:48:05.239212] [ '
                                            'DPDK EAL parameters: [2022-10-26 '
                                            '14:48:05.239218] bdevperf '
                                            '[2022-10-26 14:48:05.239223] '
                                            '--no-shconf [2022-10-26 '
                                            '14:48:05.239228] -l 0 [2022-10-26 '
                                            '14:48:05.239233] --huge-unlink '
                                            '[2022-10-26 14:48:05.239237] '
                                            '--log-level=lib.eal:6 [2022-10-26 '
                                            '14:48:05.239241] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-26 14:48:05.239246] '
                                            '--log-level=user1:6 [2022-10-26 '
                                            '14:48:05.239250] --iova-mode=pa '
                                            '[2022-10-26 14:48:05.239254] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-26 14:48:05.239258] '
                                            '--match-allocations [2022-10-26 '
                                            '14:48:05.239264] '
                                            '--file-prefix=spdk_pid3653 '
                                            '[2022-10-26 14:48:05.239268] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-26 14:48:05.361024] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-26 14:48:05.410953] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-26 14:48:05.411512] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_22.output'),
                                  'state': {'begin': 1666781394.3306637,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_aio_foo.conf '
                                                   '-q 1 -o 512 -w randread -t '
                                                   "10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.178808212280273,
                                            'end': 1666781405.509472,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_22.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_22.state')},
                       'cmd_23': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_23.output'),
                                  'state': {'begin': 1666781405.5125322,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_aio_foo.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.7970554828643799,
                                            'end': 1666781406.3095877,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_23.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_23.state')},
                       'cmd_24': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :  '
                                            '154684.51 IOPS      75.53 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :  '
                                            '154684.51 IOPS      75.53 MiB/s\n'
                                            '[2022-10-26 14:48:18.287627] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-26 14:48:18.287671] [ '
                                            'DPDK EAL parameters: [2022-10-26 '
                                            '14:48:18.287677] bdevperf '
                                            '[2022-10-26 14:48:18.287682] '
                                            '--no-shconf [2022-10-26 '
                                            '14:48:18.287686] -l 0 [2022-10-26 '
                                            '14:48:18.287691] --huge-unlink '
                                            '[2022-10-26 14:48:18.287695] '
                                            '--log-level=lib.eal:6 [2022-10-26 '
                                            '14:48:18.287700] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-26 14:48:18.287703] '
                                            '--log-level=user1:6 [2022-10-26 '
                                            '14:48:18.287708] --iova-mode=pa '
                                            '[2022-10-26 14:48:18.287712] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-26 14:48:18.287716] '
                                            '--match-allocations [2022-10-26 '
                                            '14:48:18.287721] '
                                            '--file-prefix=spdk_pid3679 '
                                            '[2022-10-26 14:48:18.287727] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-26 14:48:18.409256] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-26 14:48:18.452516] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-26 14:48:18.453072] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_24.output'),
                                  'state': {'begin': 1666781407.3719943,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_aio_foo.conf '
                                                   '-q 1 -o 512 -w randread -t '
                                                   "10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.219205141067505,
                                            'end': 1666781418.5911994,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_24.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_24.state')},
                       'cmd_25': {'output': 'rm: cannot remove '
                                            "'/tmp/spdk_bdev_xnvme_libaio.conf': "
                                            'No such file or directory\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_25.output'),
                                  'state': {'begin': 1666781418.5938334,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_xnvme_libaio.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.8456614017486572,
                                            'end': 1666781419.4394948,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_25.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_25.state')},
                       'cmd_26': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :  '
                                            '145398.43 IOPS      71.00 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :  '
                                            '145398.43 IOPS      71.00 MiB/s\n'
                                            '[2022-10-26 14:48:31.500046] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-26 14:48:31.500091] [ '
                                            'DPDK EAL parameters: [2022-10-26 '
                                            '14:48:31.500096] bdevperf '
                                            '[2022-10-26 14:48:31.500102] '
                                            '--no-shconf [2022-10-26 '
                                            '14:48:31.500106] -l 0 [2022-10-26 '
                                            '14:48:31.500111] --huge-unlink '
                                            '[2022-10-26 14:48:31.500115] '
                                            '--log-level=lib.eal:6 [2022-10-26 '
                                            '14:48:31.500119] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-26 14:48:31.500123] '
                                            '--log-level=user1:6 [2022-10-26 '
                                            '14:48:31.500128] --iova-mode=pa '
                                            '[2022-10-26 14:48:31.500132] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-26 14:48:31.500136] '
                                            '--match-allocations [2022-10-26 '
                                            '14:48:31.500140] '
                                            '--file-prefix=spdk_pid3704 '
                                            '[2022-10-26 14:48:31.500144] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-26 14:48:31.621779] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-26 14:48:31.662966] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-26 14:48:31.663515] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_26.output'),
                                  'state': {'begin': 1666781420.516107,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_xnvme_libaio.conf '
                                                   '-q 1 -o 512 -w randread -t '
                                                   "10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.241715669631958,
                                            'end': 1666781431.7578228,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_26.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_26.state')},
                       'cmd_27': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_27.output'),
                                  'state': {'begin': 1666781431.7608914,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_xnvme_libaio.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.794175386428833,
                                            'end': 1666781432.5550668,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_27.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_27.state')},
                       'cmd_28': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :  '
                                            '146366.04 IOPS      71.47 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :  '
                                            '146366.04 IOPS      71.47 MiB/s\n'
                                            '[2022-10-26 14:48:44.543718] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-26 14:48:44.543768] [ '
                                            'DPDK EAL parameters: [2022-10-26 '
                                            '14:48:44.543775] bdevperf '
                                            '[2022-10-26 14:48:44.543780] '
                                            '--no-shconf [2022-10-26 '
                                            '14:48:44.543787] -l 0 [2022-10-26 '
                                            '14:48:44.543793] --huge-unlink '
                                            '[2022-10-26 14:48:44.543799] '
                                            '--log-level=lib.eal:6 [2022-10-26 '
                                            '14:48:44.543805] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-26 14:48:44.543810] '
                                            '--log-level=user1:6 [2022-10-26 '
                                            '14:48:44.543816] --iova-mode=pa '
                                            '[2022-10-26 14:48:44.543822] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-26 14:48:44.543828] '
                                            '--match-allocations [2022-10-26 '
                                            '14:48:44.543834] '
                                            '--file-prefix=spdk_pid3729 '
                                            '[2022-10-26 14:48:44.543839] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-26 14:48:44.665842] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-26 14:48:44.707021] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-26 14:48:44.707577] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_28.output'),
                                  'state': {'begin': 1666781433.6278818,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_xnvme_libaio.conf '
                                                   '-q 1 -o 512 -w randread -t '
                                                   "10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.179730653762817,
                                            'end': 1666781444.8076124,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_28.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_28.state')},
                       'cmd_29': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_29.output'),
                                  'state': {'begin': 1666781444.81459,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_xnvme_libaio.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.8071081638336182,
                                            'end': 1666781445.6216981,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_29.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_29.state')},
                       'cmd_30': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :  '
                                            '146618.86 IOPS      71.59 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :  '
                                            '146618.86 IOPS      71.59 MiB/s\n'
                                            '[2022-10-26 14:48:57.629719] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-26 14:48:57.629762] [ '
                                            'DPDK EAL parameters: [2022-10-26 '
                                            '14:48:57.629768] bdevperf '
                                            '[2022-10-26 14:48:57.629773] '
                                            '--no-shconf [2022-10-26 '
                                            '14:48:57.629778] -l 0 [2022-10-26 '
                                            '14:48:57.629782] --huge-unlink '
                                            '[2022-10-26 14:48:57.629785] '
                                            '--log-level=lib.eal:6 [2022-10-26 '
                                            '14:48:57.629790] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-26 14:48:57.629793] '
                                            '--log-level=user1:6 [2022-10-26 '
                                            '14:48:57.629798] --iova-mode=pa '
                                            '[2022-10-26 14:48:57.629801] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-26 14:48:57.629805] '
                                            '--match-allocations [2022-10-26 '
                                            '14:48:57.629809] '
                                            '--file-prefix=spdk_pid3755 '
                                            '[2022-10-26 14:48:57.629813] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-26 14:48:57.752067] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-26 14:48:57.794987] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-26 14:48:57.795541] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_30.output'),
                                  'state': {'begin': 1666781446.7268162,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_xnvme_libaio.conf '
                                                   '-q 1 -o 512 -w randread -t '
                                                   "10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.14713454246521,
                                            'end': 1666781457.8739507,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_30.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_30.state')},
                       'cmd_31': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_31.output'),
                                  'state': {'begin': 1666781457.875226,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.8208591938018799,
                                            'end': 1666781458.6960852,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_31.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_31.state')},
                       'cmd_32': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :  '
                                            '261816.82 IOPS     127.84 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :  '
                                            '261816.82 IOPS     127.84 MiB/s\n'
                                            '[2022-10-26 14:49:10.728753] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-26 14:49:10.728804] [ '
                                            'DPDK EAL parameters: [2022-10-26 '
                                            '14:49:10.728810] bdevperf '
                                            '[2022-10-26 14:49:10.728814] '
                                            '--no-shconf [2022-10-26 '
                                            '14:49:10.728818] -l 0 [2022-10-26 '
                                            '14:49:10.728823] --huge-unlink '
                                            '[2022-10-26 14:49:10.728827] '
                                            '--log-level=lib.eal:6 [2022-10-26 '
                                            '14:49:10.728831] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-26 14:49:10.728835] '
                                            '--log-level=user1:6 [2022-10-26 '
                                            '14:49:10.728839] --iova-mode=pa '
                                            '[2022-10-26 14:49:10.728843] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-26 14:49:10.728847] '
                                            '--match-allocations [2022-10-26 '
                                            '14:49:10.728851] '
                                            '--file-prefix=spdk_pid3780 '
                                            '[2022-10-26 14:49:10.728857] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-26 14:49:10.850519] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-26 14:49:10.894989] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-26 14:49:10.895537] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_32.output'),
                                  'state': {'begin': 1666781459.750432,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
                                                   '-q 2 -o 512 -w randread -t '
                                                   "10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.17123556137085,
                                            'end': 1666781470.9216676,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_32.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_32.state')},
                       'cmd_33': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_33.output'),
                                  'state': {'begin': 1666781470.9237893,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.898444414138794,
                                            'end': 1666781471.8222337,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_33.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_33.state')},
                       'cmd_34': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :  '
                                            '261099.59 IOPS     127.49 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :  '
                                            '261099.59 IOPS     127.49 MiB/s\n'
                                            '[2022-10-26 14:49:23.852529] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-26 14:49:23.852573] [ '
                                            'DPDK EAL parameters: [2022-10-26 '
                                            '14:49:23.852579] bdevperf '
                                            '[2022-10-26 14:49:23.852583] '
                                            '--no-shconf [2022-10-26 '
                                            '14:49:23.852589] -l 0 [2022-10-26 '
                                            '14:49:23.852594] --huge-unlink '
                                            '[2022-10-26 14:49:23.852599] '
                                            '--log-level=lib.eal:6 [2022-10-26 '
                                            '14:49:23.852603] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-26 14:49:23.852608] '
                                            '--log-level=user1:6 [2022-10-26 '
                                            '14:49:23.852612] --iova-mode=pa '
                                            '[2022-10-26 14:49:23.852616] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-26 14:49:23.852621] '
                                            '--match-allocations [2022-10-26 '
                                            '14:49:23.852626] '
                                            '--file-prefix=spdk_pid3805 '
                                            '[2022-10-26 14:49:23.852632] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-26 14:49:23.974359] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-26 14:49:24.027041] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-26 14:49:24.027602] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_34.output'),
                                  'state': {'begin': 1666781472.9142237,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
                                                   '-q 2 -o 512 -w randread -t '
                                                   "10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.13983416557312,
                                            'end': 1666781484.0540578,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_34.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_34.state')},
                       'cmd_35': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_35.output'),
                                  'state': {'begin': 1666781484.0572314,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.8143582344055176,
                                            'end': 1666781484.8715897,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_35.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_35.state')},
                       'cmd_36': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :  '
                                            '260889.10 IOPS     127.39 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :  '
                                            '260889.10 IOPS     127.39 MiB/s\n'
                                            '[2022-10-26 14:49:36.826143] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-26 14:49:36.826185] [ '
                                            'DPDK EAL parameters: [2022-10-26 '
                                            '14:49:36.826191] bdevperf '
                                            '[2022-10-26 14:49:36.826195] '
                                            '--no-shconf [2022-10-26 '
                                            '14:49:36.826200] -l 0 [2022-10-26 '
                                            '14:49:36.826204] --huge-unlink '
                                            '[2022-10-26 14:49:36.826209] '
                                            '--log-level=lib.eal:6 [2022-10-26 '
                                            '14:49:36.826213] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-26 14:49:36.826218] '
                                            '--log-level=user1:6 [2022-10-26 '
                                            '14:49:36.826222] --iova-mode=pa '
                                            '[2022-10-26 14:49:36.826226] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-26 14:49:36.826230] '
                                            '--match-allocations [2022-10-26 '
                                            '14:49:36.826235] '
                                            '--file-prefix=spdk_pid3830 '
                                            '[2022-10-26 14:49:36.826239] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-26 14:49:36.947892] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-26 14:49:36.991044] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-26 14:49:36.991602] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_36.output'),
                                  'state': {'begin': 1666781485.9342191,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
                                                   '-q 2 -o 512 -w randread -t '
                                                   "10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.135831117630005,
                                            'end': 1666781497.0700502,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_36.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_36.state')},
                       'cmd_37': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_37.output'),
                                  'state': {'begin': 1666781497.072821,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_uring_foo.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.8443262577056885,
                                            'end': 1666781497.9171472,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_37.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_37.state')},
                       'cmd_38': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :  '
                                            '230826.32 IOPS     112.71 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :  '
                                            '230826.32 IOPS     112.71 MiB/s\n'
                                            '[2022-10-26 14:49:49.866229] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-26 14:49:49.866273] [ '
                                            'DPDK EAL parameters: [2022-10-26 '
                                            '14:49:49.866279] bdevperf '
                                            '[2022-10-26 14:49:49.866283] '
                                            '--no-shconf [2022-10-26 '
                                            '14:49:49.866288] -l 0 [2022-10-26 '
                                            '14:49:49.866293] --huge-unlink '
                                            '[2022-10-26 14:49:49.866298] '
                                            '--log-level=lib.eal:6 [2022-10-26 '
                                            '14:49:49.866303] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-26 14:49:49.866307] '
                                            '--log-level=user1:6 [2022-10-26 '
                                            '14:49:49.866311] --iova-mode=pa '
                                            '[2022-10-26 14:49:49.866316] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-26 14:49:49.866321] '
                                            '--match-allocations [2022-10-26 '
                                            '14:49:49.866325] '
                                            '--file-prefix=spdk_pid3855 '
                                            '[2022-10-26 14:49:49.866330] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-26 14:49:49.987927] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-26 14:49:50.031010] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-26 14:49:50.031564] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_38.output'),
                                  'state': {'begin': 1666781498.9793825,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_uring_foo.conf '
                                                   '-q 2 -o 512 -w randread -t '
                                                   "10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.064125537872314,
                                            'end': 1666781510.043508,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_38.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_38.state')},
                       'cmd_39': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_39.output'),
                                  'state': {'begin': 1666781510.046105,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_uring_foo.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.7939174175262451,
                                            'end': 1666781510.8400223,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_39.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_39.state')},
                       'cmd_40': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :  '
                                            '229739.44 IOPS     112.18 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :  '
                                            '229739.44 IOPS     112.18 MiB/s\n'
                                            '[2022-10-26 14:50:02.780961] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-26 14:50:02.781004] [ '
                                            'DPDK EAL parameters: [2022-10-26 '
                                            '14:50:02.781010] bdevperf '
                                            '[2022-10-26 14:50:02.781015] '
                                            '--no-shconf [2022-10-26 '
                                            '14:50:02.781020] -l 0 [2022-10-26 '
                                            '14:50:02.781026] --huge-unlink '
                                            '[2022-10-26 14:50:02.781030] '
                                            '--log-level=lib.eal:6 [2022-10-26 '
                                            '14:50:02.781034] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-26 14:50:02.781038] '
                                            '--log-level=user1:6 [2022-10-26 '
                                            '14:50:02.781042] --iova-mode=pa '
                                            '[2022-10-26 14:50:02.781046] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-26 14:50:02.781050] '
                                            '--match-allocations [2022-10-26 '
                                            '14:50:02.781060] '
                                            '--file-prefix=spdk_pid3880 '
                                            '[2022-10-26 14:50:02.781064] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-26 14:50:02.902700] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-26 14:50:02.946942] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-26 14:50:02.947497] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_40.output'),
                                  'state': {'begin': 1666781511.8811512,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_uring_foo.conf '
                                                   '-q 2 -o 512 -w randread -t '
                                                   "10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.078004837036133,
                                            'end': 1666781522.959156,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_40.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_40.state')},
                       'cmd_41': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_41.output'),
                                  'state': {'begin': 1666781522.9622078,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_uring_foo.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.8215856552124023,
                                            'end': 1666781523.7837934,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_41.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_41.state')},
                       'cmd_42': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :  '
                                            '231226.96 IOPS     112.90 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :  '
                                            '231226.96 IOPS     112.90 MiB/s\n'
                                            '[2022-10-26 14:50:15.813398] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-26 14:50:15.813439] [ '
                                            'DPDK EAL parameters: [2022-10-26 '
                                            '14:50:15.813445] bdevperf '
                                            '[2022-10-26 14:50:15.813450] '
                                            '--no-shconf [2022-10-26 '
                                            '14:50:15.813455] -l 0 [2022-10-26 '
                                            '14:50:15.813460] --huge-unlink '
                                            '[2022-10-26 14:50:15.813464] '
                                            '--log-level=lib.eal:6 [2022-10-26 '
                                            '14:50:15.813470] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-26 14:50:15.813474] '
                                            '--log-level=user1:6 [2022-10-26 '
                                            '14:50:15.813478] --iova-mode=pa '
                                            '[2022-10-26 14:50:15.813482] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-26 14:50:15.813487] '
                                            '--match-allocations [2022-10-26 '
                                            '14:50:15.813491] '
                                            '--file-prefix=spdk_pid3906 '
                                            '[2022-10-26 14:50:15.813495] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-26 14:50:15.935094] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-26 14:50:15.978913] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-26 14:50:15.979453] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_42.output'),
                                  'state': {'begin': 1666781524.9029205,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_uring_foo.conf '
                                                   '-q 2 -o 512 -w randread -t '
                                                   "10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.087957382202148,
                                            'end': 1666781535.9908779,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_42.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_42.state')},
                       'cmd_43': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_43.output'),
                                  'state': {'begin': 1666781535.9935784,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_xnvme_io_uring.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.8295695781707764,
                                            'end': 1666781536.823148,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_43.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_43.state')},
                       'cmd_44': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :  '
                                            '282909.29 IOPS     138.14 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :  '
                                            '282909.29 IOPS     138.14 MiB/s\n'
                                            '[2022-10-26 14:50:28.780387] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-26 14:50:28.780434] [ '
                                            'DPDK EAL parameters: [2022-10-26 '
                                            '14:50:28.780440] bdevperf '
                                            '[2022-10-26 14:50:28.780445] '
                                            '--no-shconf [2022-10-26 '
                                            '14:50:28.780449] -l 0 [2022-10-26 '
                                            '14:50:28.780453] --huge-unlink '
                                            '[2022-10-26 14:50:28.780458] '
                                            '--log-level=lib.eal:6 [2022-10-26 '
                                            '14:50:28.780463] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-26 14:50:28.780467] '
                                            '--log-level=user1:6 [2022-10-26 '
                                            '14:50:28.780471] --iova-mode=pa '
                                            '[2022-10-26 14:50:28.780475] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-26 14:50:28.780479] '
                                            '--match-allocations [2022-10-26 '
                                            '14:50:28.780485] '
                                            '--file-prefix=spdk_pid3931 '
                                            '[2022-10-26 14:50:28.780491] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-26 14:50:28.902341] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-26 14:50:28.948483] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-26 14:50:28.949036] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_44.output'),
                                  'state': {'begin': 1666781537.8824975,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_xnvme_io_uring.conf '
                                                   '-q 2 -o 512 -w randread -t '
                                                   "10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.088404178619385,
                                            'end': 1666781548.9709017,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_44.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_44.state')},
                       'cmd_45': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_45.output'),
                                  'state': {'begin': 1666781548.9792178,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_xnvme_io_uring.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.800908088684082,
                                            'end': 1666781549.7801259,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_45.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_45.state')},
                       'cmd_46': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :  '
                                            '283679.60 IOPS     138.52 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :  '
                                            '283679.60 IOPS     138.52 MiB/s\n'
                                            '[2022-10-26 14:50:41.753061] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-26 14:50:41.753108] [ '
                                            'DPDK EAL parameters: [2022-10-26 '
                                            '14:50:41.753113] bdevperf '
                                            '[2022-10-26 14:50:41.753119] '
                                            '--no-shconf [2022-10-26 '
                                            '14:50:41.753123] -l 0 [2022-10-26 '
                                            '14:50:41.753128] --huge-unlink '
                                            '[2022-10-26 14:50:41.753132] '
                                            '--log-level=lib.eal:6 [2022-10-26 '
                                            '14:50:41.753138] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-26 14:50:41.753142] '
                                            '--log-level=user1:6 [2022-10-26 '
                                            '14:50:41.753146] --iova-mode=pa '
                                            '[2022-10-26 14:50:41.753150] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-26 14:50:41.753154] '
                                            '--match-allocations [2022-10-26 '
                                            '14:50:41.753160] '
                                            '--file-prefix=spdk_pid3957 '
                                            '[2022-10-26 14:50:41.753164] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-26 14:50:41.874953] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-26 14:50:41.915040] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-26 14:50:41.915590] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_46.output'),
                                  'state': {'begin': 1666781550.867793,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_xnvme_io_uring.conf '
                                                   '-q 2 -o 512 -w randread -t '
                                                   "10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.061692476272583,
                                            'end': 1666781561.9294856,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_46.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_46.state')},
                       'cmd_47': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_47.output'),
                                  'state': {'begin': 1666781561.9321043,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_xnvme_io_uring.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.7853102684020996,
                                            'end': 1666781562.7174146,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_47.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_47.state')},
                       'cmd_48': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :  '
                                            '283155.55 IOPS     138.26 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :  '
                                            '283155.55 IOPS     138.26 MiB/s\n'
                                            '[2022-10-26 14:50:54.724079] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-26 14:50:54.724121] [ '
                                            'DPDK EAL parameters: [2022-10-26 '
                                            '14:50:54.724127] bdevperf '
                                            '[2022-10-26 14:50:54.724132] '
                                            '--no-shconf [2022-10-26 '
                                            '14:50:54.724137] -l 0 [2022-10-26 '
                                            '14:50:54.724141] --huge-unlink '
                                            '[2022-10-26 14:50:54.724146] '
                                            '--log-level=lib.eal:6 [2022-10-26 '
                                            '14:50:54.724149] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-26 14:50:54.724154] '
                                            '--log-level=user1:6 [2022-10-26 '
                                            '14:50:54.724157] --iova-mode=pa '
                                            '[2022-10-26 14:50:54.724161] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-26 14:50:54.724165] '
                                            '--match-allocations [2022-10-26 '
                                            '14:50:54.724169] '
                                            '--file-prefix=spdk_pid3984 '
                                            '[2022-10-26 14:50:54.724175] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-26 14:50:54.845795] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-26 14:50:54.887066] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-26 14:50:54.887623] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_48.output'),
                                  'state': {'begin': 1666781563.814571,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_xnvme_io_uring.conf '
                                                   '-q 2 -o 512 -w randread -t '
                                                   "10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.088241577148438,
                                            'end': 1666781574.9028125,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_48.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_48.state')},
                       'cmd_49': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_49.output'),
                                  'state': {'begin': 1666781574.905429,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_aio_foo.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.8327929973602295,
                                            'end': 1666781575.738222,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_49.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_49.state')},
                       'cmd_50': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :  '
                                            '281466.49 IOPS     137.43 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :  '
                                            '281466.49 IOPS     137.43 MiB/s\n'
                                            '[2022-10-26 14:51:07.680050] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-26 14:51:07.680094] [ '
                                            'DPDK EAL parameters: [2022-10-26 '
                                            '14:51:07.680101] bdevperf '
                                            '[2022-10-26 14:51:07.680106] '
                                            '--no-shconf [2022-10-26 '
                                            '14:51:07.680111] -l 0 [2022-10-26 '
                                            '14:51:07.680118] --huge-unlink '
                                            '[2022-10-26 14:51:07.680126] '
                                            '--log-level=lib.eal:6 [2022-10-26 '
                                            '14:51:07.680132] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-26 14:51:07.680137] '
                                            '--log-level=user1:6 [2022-10-26 '
                                            '14:51:07.680142] --iova-mode=pa '
                                            '[2022-10-26 14:51:07.680149] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-26 14:51:07.680154] '
                                            '--match-allocations [2022-10-26 '
                                            '14:51:07.680160] '
                                            '--file-prefix=spdk_pid4010 '
                                            '[2022-10-26 14:51:07.680167] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-26 14:51:07.801780] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-26 14:51:07.839036] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-26 14:51:07.839594] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_50.output'),
                                  'state': {'begin': 1666781576.7971725,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_aio_foo.conf '
                                                   '-q 2 -o 512 -w randread -t '
                                                   "10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.1195068359375,
                                            'end': 1666781587.9166794,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_50.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_50.state')},
                       'cmd_51': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_51.output'),
                                  'state': {'begin': 1666781587.9197438,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_aio_foo.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.7945308685302734,
                                            'end': 1666781588.7142746,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_51.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_51.state')},
                       'cmd_52': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :  '
                                            '280747.16 IOPS     137.08 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :  '
                                            '280747.16 IOPS     137.08 MiB/s\n'
                                            '[2022-10-26 14:51:20.723914] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-26 14:51:20.723958] [ '
                                            'DPDK EAL parameters: [2022-10-26 '
                                            '14:51:20.723964] bdevperf '
                                            '[2022-10-26 14:51:20.723968] '
                                            '--no-shconf [2022-10-26 '
                                            '14:51:20.723973] -l 0 [2022-10-26 '
                                            '14:51:20.723979] --huge-unlink '
                                            '[2022-10-26 14:51:20.723983] '
                                            '--log-level=lib.eal:6 [2022-10-26 '
                                            '14:51:20.723988] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-26 14:51:20.723992] '
                                            '--log-level=user1:6 [2022-10-26 '
                                            '14:51:20.723997] --iova-mode=pa '
                                            '[2022-10-26 14:51:20.724001] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-26 14:51:20.724004] '
                                            '--match-allocations [2022-10-26 '
                                            '14:51:20.724009] '
                                            '--file-prefix=spdk_pid4036 '
                                            '[2022-10-26 14:51:20.724014] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-26 14:51:20.845500] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-26 14:51:20.895019] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-26 14:51:20.895578] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_52.output'),
                                  'state': {'begin': 1666781589.80861,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_aio_foo.conf '
                                                   '-q 2 -o 512 -w randread -t '
                                                   "10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.155845403671265,
                                            'end': 1666781600.9644554,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_52.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_52.state')},
                       'cmd_53': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_53.output'),
                                  'state': {'begin': 1666781600.9671118,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_aio_foo.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.7982139587402344,
                                            'end': 1666781601.7653258,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_53.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_53.state')},
                       'cmd_54': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :  '
                                            '280777.96 IOPS     137.10 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :  '
                                            '280777.96 IOPS     137.10 MiB/s\n'
                                            '[2022-10-26 14:51:33.686533] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-26 14:51:33.686578] [ '
                                            'DPDK EAL parameters: [2022-10-26 '
                                            '14:51:33.686584] bdevperf '
                                            '[2022-10-26 14:51:33.686589] '
                                            '--no-shconf [2022-10-26 '
                                            '14:51:33.686594] -l 0 [2022-10-26 '
                                            '14:51:33.686598] --huge-unlink '
                                            '[2022-10-26 14:51:33.686603] '
                                            '--log-level=lib.eal:6 [2022-10-26 '
                                            '14:51:33.686608] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-26 14:51:33.686612] '
                                            '--log-level=user1:6 [2022-10-26 '
                                            '14:51:33.686616] --iova-mode=pa '
                                            '[2022-10-26 14:51:33.686620] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-26 14:51:33.686625] '
                                            '--match-allocations [2022-10-26 '
                                            '14:51:33.686630] '
                                            '--file-prefix=spdk_pid4063 '
                                            '[2022-10-26 14:51:33.686635] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-26 14:51:33.808399] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-26 14:51:33.851003] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-26 14:51:33.851560] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_54.output'),
                                  'state': {'begin': 1666781602.806074,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_aio_foo.conf '
                                                   '-q 2 -o 512 -w randread -t '
                                                   "10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.11859655380249,
                                            'end': 1666781613.9246705,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_54.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_54.state')},
                       'cmd_55': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_55.output'),
                                  'state': {'begin': 1666781613.9272351,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_xnvme_libaio.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.8031032085418701,
                                            'end': 1666781614.7303383,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_55.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_55.state')},
                       'cmd_56': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :  '
                                            '267085.41 IOPS     130.41 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :  '
                                            '267085.41 IOPS     130.41 MiB/s\n'
                                            '[2022-10-26 14:51:46.743969] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-26 14:51:46.744011] [ '
                                            'DPDK EAL parameters: [2022-10-26 '
                                            '14:51:46.744017] bdevperf '
                                            '[2022-10-26 14:51:46.744021] '
                                            '--no-shconf [2022-10-26 '
                                            '14:51:46.744026] -l 0 [2022-10-26 '
                                            '14:51:46.744030] --huge-unlink '
                                            '[2022-10-26 14:51:46.744034] '
                                            '--log-level=lib.eal:6 [2022-10-26 '
                                            '14:51:46.744038] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-26 14:51:46.744044] '
                                            '--log-level=user1:6 [2022-10-26 '
                                            '14:51:46.744048] --iova-mode=pa '
                                            '[2022-10-26 14:51:46.744053] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-26 14:51:46.744058] '
                                            '--match-allocations [2022-10-26 '
                                            '14:51:46.744064] '
                                            '--file-prefix=spdk_pid4090 '
                                            '[2022-10-26 14:51:46.744068] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-26 14:51:46.866027] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-26 14:51:46.903064] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-26 14:51:46.903629] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_56.output'),
                                  'state': {'begin': 1666781615.849239,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_xnvme_libaio.conf '
                                                   '-q 2 -o 512 -w randread -t '
                                                   "10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.135571241378784,
                                            'end': 1666781626.9848104,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_56.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_56.state')},
                       'cmd_57': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_57.output'),
                                  'state': {'begin': 1666781626.9873095,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_xnvme_libaio.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.8279106616973877,
                                            'end': 1666781627.81522,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_57.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_57.state')},
                       'cmd_58': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :  '
                                            '267177.11 IOPS     130.46 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :  '
                                            '267177.11 IOPS     130.46 MiB/s\n'
                                            '[2022-10-26 14:51:59.778212] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-26 14:51:59.778256] [ '
                                            'DPDK EAL parameters: [2022-10-26 '
                                            '14:51:59.778262] bdevperf '
                                            '[2022-10-26 14:51:59.778267] '
                                            '--no-shconf [2022-10-26 '
                                            '14:51:59.778271] -l 0 [2022-10-26 '
                                            '14:51:59.778276] --huge-unlink '
                                            '[2022-10-26 14:51:59.778281] '
                                            '--log-level=lib.eal:6 [2022-10-26 '
                                            '14:51:59.778285] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-26 14:51:59.778291] '
                                            '--log-level=user1:6 [2022-10-26 '
                                            '14:51:59.778295] --iova-mode=pa '
                                            '[2022-10-26 14:51:59.778299] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-26 14:51:59.778303] '
                                            '--match-allocations [2022-10-26 '
                                            '14:51:59.778307] '
                                            '--file-prefix=spdk_pid4115 '
                                            '[2022-10-26 14:51:59.778313] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-26 14:51:59.900002] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-26 14:51:59.951001] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-26 14:51:59.951562] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_58.output'),
                                  'state': {'begin': 1666781628.8598037,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_xnvme_libaio.conf '
                                                   '-q 2 -o 512 -w randread -t '
                                                   "10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.180998086929321,
                                            'end': 1666781640.0408018,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_58.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_58.state')},
                       'cmd_59': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_59.output'),
                                  'state': {'begin': 1666781640.04412,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_xnvme_libaio.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.8200516700744629,
                                            'end': 1666781640.8641717,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_59.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_59.state')},
                       'cmd_60': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :  '
                                            '266611.68 IOPS     130.18 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :  '
                                            '266611.68 IOPS     130.18 MiB/s\n'
                                            '[2022-10-26 14:52:12.820906] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-26 14:52:12.820957] [ '
                                            'DPDK EAL parameters: [2022-10-26 '
                                            '14:52:12.820965] bdevperf '
                                            '[2022-10-26 14:52:12.820972] '
                                            '--no-shconf [2022-10-26 '
                                            '14:52:12.820978] -l 0 [2022-10-26 '
                                            '14:52:12.820984] --huge-unlink '
                                            '[2022-10-26 14:52:12.820990] '
                                            '--log-level=lib.eal:6 [2022-10-26 '
                                            '14:52:12.820997] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-26 14:52:12.821003] '
                                            '--log-level=user1:6 [2022-10-26 '
                                            '14:52:12.821011] --iova-mode=pa '
                                            '[2022-10-26 14:52:12.821019] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-26 14:52:12.821025] '
                                            '--match-allocations [2022-10-26 '
                                            '14:52:12.821032] '
                                            '--file-prefix=spdk_pid4140 '
                                            '[2022-10-26 14:52:12.821039] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-26 14:52:12.942842] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-26 14:52:12.987011] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-26 14:52:12.987571] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_60.output'),
                                  'state': {'begin': 1666781641.9165711,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_xnvme_libaio.conf '
                                                   '-q 2 -o 512 -w randread -t '
                                                   "10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.18920350074768,
                                            'end': 1666781653.1057746,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_60.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_60.state')},
                       'cmd_61': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_61.output'),
                                  'state': {'begin': 1666781653.1083791,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.8270316123962402,
                                            'end': 1666781653.9354107,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_61.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_61.state')},
                       'cmd_62': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :  '
                                            '409829.09 IOPS     200.11 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :  '
                                            '409829.09 IOPS     200.11 MiB/s\n'
                                            '[2022-10-26 14:52:25.997001] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-26 14:52:25.997050] [ '
                                            'DPDK EAL parameters: [2022-10-26 '
                                            '14:52:25.997056] bdevperf '
                                            '[2022-10-26 14:52:25.997060] '
                                            '--no-shconf [2022-10-26 '
                                            '14:52:25.997064] -l 0 [2022-10-26 '
                                            '14:52:25.997069] --huge-unlink '
                                            '[2022-10-26 14:52:25.997074] '
                                            '--log-level=lib.eal:6 [2022-10-26 '
                                            '14:52:25.997078] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-26 14:52:25.997083] '
                                            '--log-level=user1:6 [2022-10-26 '
                                            '14:52:25.997087] --iova-mode=pa '
                                            '[2022-10-26 14:52:25.997091] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-26 14:52:25.997095] '
                                            '--match-allocations [2022-10-26 '
                                            '14:52:25.997100] '
                                            '--file-prefix=spdk_pid4165 '
                                            '[2022-10-26 14:52:25.997105] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-26 14:52:26.118822] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-26 14:52:26.162992] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-26 14:52:26.163553] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_62.output'),
                                  'state': {'begin': 1666781655.1041815,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
                                                   '-q 4 -o 512 -w randread -t '
                                                   "10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.088993072509766,
                                            'end': 1666781666.1931746,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_62.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_62.state')},
                       'cmd_63': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_63.output'),
                                  'state': {'begin': 1666781666.1958718,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.8419077396392822,
                                            'end': 1666781667.0377796,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_63.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_63.state')},
                       'cmd_64': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :  '
                                            '453036.84 IOPS     221.21 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :  '
                                            '453036.84 IOPS     221.21 MiB/s\n'
                                            '[2022-10-26 14:52:39.039669] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-26 14:52:39.039712] [ '
                                            'DPDK EAL parameters: [2022-10-26 '
                                            '14:52:39.039717] bdevperf '
                                            '[2022-10-26 14:52:39.039722] '
                                            '--no-shconf [2022-10-26 '
                                            '14:52:39.039726] -l 0 [2022-10-26 '
                                            '14:52:39.039731] --huge-unlink '
                                            '[2022-10-26 14:52:39.039736] '
                                            '--log-level=lib.eal:6 [2022-10-26 '
                                            '14:52:39.039740] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-26 14:52:39.039745] '
                                            '--log-level=user1:6 [2022-10-26 '
                                            '14:52:39.039749] --iova-mode=pa '
                                            '[2022-10-26 14:52:39.039752] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-26 14:52:39.039757] '
                                            '--match-allocations [2022-10-26 '
                                            '14:52:39.039761] '
                                            '--file-prefix=spdk_pid4190 '
                                            '[2022-10-26 14:52:39.039766] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-26 14:52:39.161430] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-26 14:52:39.203055] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-26 14:52:39.203611] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_64.output'),
                                  'state': {'begin': 1666781668.1494107,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
                                                   '-q 4 -o 512 -w randread -t '
                                                   "10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.081124782562256,
                                            'end': 1666781679.2305355,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_64.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_64.state')},
                       'cmd_65': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_65.output'),
                                  'state': {'begin': 1666781679.2331562,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 1.0669896602630615,
                                            'end': 1666781680.3001459,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_65.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_65.state')},
                       'cmd_66': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :  '
                                            '493618.05 IOPS     241.02 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :  '
                                            '493618.05 IOPS     241.02 MiB/s\n'
                                            '[2022-10-26 14:52:52.548321] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-26 14:52:52.548367] [ '
                                            'DPDK EAL parameters: [2022-10-26 '
                                            '14:52:52.548372] bdevperf '
                                            '[2022-10-26 14:52:52.548377] '
                                            '--no-shconf [2022-10-26 '
                                            '14:52:52.548381] -l 0 [2022-10-26 '
                                            '14:52:52.548385] --huge-unlink '
                                            '[2022-10-26 14:52:52.548390] '
                                            '--log-level=lib.eal:6 [2022-10-26 '
                                            '14:52:52.548394] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-26 14:52:52.548398] '
                                            '--log-level=user1:6 [2022-10-26 '
                                            '14:52:52.548402] --iova-mode=pa '
                                            '[2022-10-26 14:52:52.548406] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-26 14:52:52.548410] '
                                            '--match-allocations [2022-10-26 '
                                            '14:52:52.548415] '
                                            '--file-prefix=spdk_pid4215 '
                                            '[2022-10-26 14:52:52.548422] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-26 14:52:52.669991] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-26 14:52:52.711005] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-26 14:52:52.711571] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_66.output'),
                                  'state': {'begin': 1666781681.5408564,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
                                                   '-q 4 -o 512 -w randread -t '
                                                   "10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.20149278640747,
                                            'end': 1666781692.7423491,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_66.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_66.state')},
                       'cmd_67': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_67.output'),
                                  'state': {'begin': 1666781692.745624,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_uring_foo.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.7977368831634521,
                                            'end': 1666781693.543361,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_67.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_67.state')},
                       'cmd_68': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :  '
                                            '284288.83 IOPS     138.81 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :  '
                                            '284288.83 IOPS     138.81 MiB/s\n'
                                            '[2022-10-26 14:53:05.541424] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-26 14:53:05.541465] [ '
                                            'DPDK EAL parameters: [2022-10-26 '
                                            '14:53:05.541471] bdevperf '
                                            '[2022-10-26 14:53:05.541475] '
                                            '--no-shconf [2022-10-26 '
                                            '14:53:05.541480] -l 0 [2022-10-26 '
                                            '14:53:05.541484] --huge-unlink '
                                            '[2022-10-26 14:53:05.541488] '
                                            '--log-level=lib.eal:6 [2022-10-26 '
                                            '14:53:05.541493] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-26 14:53:05.541497] '
                                            '--log-level=user1:6 [2022-10-26 '
                                            '14:53:05.541501] --iova-mode=pa '
                                            '[2022-10-26 14:53:05.541506] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-26 14:53:05.541511] '
                                            '--match-allocations [2022-10-26 '
                                            '14:53:05.541515] '
                                            '--file-prefix=spdk_pid4240 '
                                            '[2022-10-26 14:53:05.541520] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-26 14:53:05.663378] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-26 14:53:05.706987] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-26 14:53:05.707569] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_68.output'),
                                  'state': {'begin': 1666781694.647939,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_uring_foo.conf '
                                                   '-q 4 -o 512 -w randread -t '
                                                   "10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.072965621948242,
                                            'end': 1666781705.7209046,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_68.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_68.state')},
                       'cmd_69': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_69.output'),
                                  'state': {'begin': 1666781705.7281647,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_uring_foo.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.7996237277984619,
                                            'end': 1666781706.5277884,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_69.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_69.state')},
                       'cmd_70': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :  '
                                            '283826.39 IOPS     138.59 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :  '
                                            '283826.39 IOPS     138.59 MiB/s\n'
                                            '[2022-10-26 14:53:18.550947] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-26 14:53:18.550994] [ '
                                            'DPDK EAL parameters: [2022-10-26 '
                                            '14:53:18.551000] bdevperf '
                                            '[2022-10-26 14:53:18.551004] '
                                            '--no-shconf [2022-10-26 '
                                            '14:53:18.551009] -l 0 [2022-10-26 '
                                            '14:53:18.551013] --huge-unlink '
                                            '[2022-10-26 14:53:18.551018] '
                                            '--log-level=lib.eal:6 [2022-10-26 '
                                            '14:53:18.551022] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-26 14:53:18.551027] '
                                            '--log-level=user1:6 [2022-10-26 '
                                            '14:53:18.551031] --iova-mode=pa '
                                            '[2022-10-26 14:53:18.551035] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-26 14:53:18.551039] '
                                            '--match-allocations [2022-10-26 '
                                            '14:53:18.551043] '
                                            '--file-prefix=spdk_pid4266 '
                                            '[2022-10-26 14:53:18.551048] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-26 14:53:18.672613] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-26 14:53:18.707088] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-26 14:53:18.707639] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_70.output'),
                                  'state': {'begin': 1666781707.6440644,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_uring_foo.conf '
                                                   '-q 4 -o 512 -w randread -t '
                                                   "10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.078567504882812,
                                            'end': 1666781718.722632,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_70.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_70.state')},
                       'cmd_71': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_71.output'),
                                  'state': {'begin': 1666781718.7253253,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_uring_foo.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.8233606815338135,
                                            'end': 1666781719.548686,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_71.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_71.state')},
                       'cmd_72': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :  '
                                            '283493.26 IOPS     138.42 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :  '
                                            '283493.26 IOPS     138.42 MiB/s\n'
                                            '[2022-10-26 14:53:31.638209] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-26 14:53:31.638256] [ '
                                            'DPDK EAL parameters: [2022-10-26 '
                                            '14:53:31.638262] bdevperf '
                                            '[2022-10-26 14:53:31.638266] '
                                            '--no-shconf [2022-10-26 '
                                            '14:53:31.638271] -l 0 [2022-10-26 '
                                            '14:53:31.638276] --huge-unlink '
                                            '[2022-10-26 14:53:31.638280] '
                                            '--log-level=lib.eal:6 [2022-10-26 '
                                            '14:53:31.638286] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-26 14:53:31.638290] '
                                            '--log-level=user1:6 [2022-10-26 '
                                            '14:53:31.638294] --iova-mode=pa '
                                            '[2022-10-26 14:53:31.638299] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-26 14:53:31.638304] '
                                            '--match-allocations [2022-10-26 '
                                            '14:53:31.638308] '
                                            '--file-prefix=spdk_pid4291 '
                                            '[2022-10-26 14:53:31.638313] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-26 14:53:31.760182] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-26 14:53:31.803079] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-26 14:53:31.803648] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_72.output'),
                                  'state': {'begin': 1666781720.663,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_uring_foo.conf '
                                                   '-q 4 -o 512 -w randread -t '
                                                   "10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.16416072845459,
                                            'end': 1666781731.8271608,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_72.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_72.state')},
                       'cmd_73': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_73.output'),
                                  'state': {'begin': 1666781731.829704,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_xnvme_io_uring.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.8839006423950195,
                                            'end': 1666781732.7136047,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_73.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_73.state')},
                       'cmd_74': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :  '
                                            '364445.67 IOPS     177.95 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :  '
                                            '364445.67 IOPS     177.95 MiB/s\n'
                                            '[2022-10-26 14:53:44.673297] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-26 14:53:44.673344] [ '
                                            'DPDK EAL parameters: [2022-10-26 '
                                            '14:53:44.673349] bdevperf '
                                            '[2022-10-26 14:53:44.673354] '
                                            '--no-shconf [2022-10-26 '
                                            '14:53:44.673358] -l 0 [2022-10-26 '
                                            '14:53:44.673363] --huge-unlink '
                                            '[2022-10-26 14:53:44.673368] '
                                            '--log-level=lib.eal:6 [2022-10-26 '
                                            '14:53:44.673373] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-26 14:53:44.673377] '
                                            '--log-level=user1:6 [2022-10-26 '
                                            '14:53:44.673381] --iova-mode=pa '
                                            '[2022-10-26 14:53:44.673386] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-26 14:53:44.673392] '
                                            '--match-allocations [2022-10-26 '
                                            '14:53:44.673396] '
                                            '--file-prefix=spdk_pid4316 '
                                            '[2022-10-26 14:53:44.673401] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-26 14:53:44.795838] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-26 14:53:44.839049] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-26 14:53:44.839650] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_74.output'),
                                  'state': {'begin': 1666781733.7894955,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_xnvme_io_uring.conf '
                                                   '-q 4 -o 512 -w randread -t '
                                                   "10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.076178550720215,
                                            'end': 1666781744.865674,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_74.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_74.state')},
                       'cmd_75': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_75.output'),
                                  'state': {'begin': 1666781744.8684454,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_xnvme_io_uring.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.8356173038482666,
                                            'end': 1666781745.7040627,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_75.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_75.state')},
                       'cmd_76': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :  '
                                            '366439.44 IOPS     178.93 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :  '
                                            '366439.44 IOPS     178.93 MiB/s\n'
                                            '[2022-10-26 14:53:57.645285] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-26 14:53:57.645331] [ '
                                            'DPDK EAL parameters: [2022-10-26 '
                                            '14:53:57.645337] bdevperf '
                                            '[2022-10-26 14:53:57.645342] '
                                            '--no-shconf [2022-10-26 '
                                            '14:53:57.645347] -l 0 [2022-10-26 '
                                            '14:53:57.645352] --huge-unlink '
                                            '[2022-10-26 14:53:57.645356] '
                                            '--log-level=lib.eal:6 [2022-10-26 '
                                            '14:53:57.645361] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-26 14:53:57.645366] '
                                            '--log-level=user1:6 [2022-10-26 '
                                            '14:53:57.645371] --iova-mode=pa '
                                            '[2022-10-26 14:53:57.645376] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-26 14:53:57.645381] '
                                            '--match-allocations [2022-10-26 '
                                            '14:53:57.645385] '
                                            '--file-prefix=spdk_pid4342 '
                                            '[2022-10-26 14:53:57.645390] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-26 14:53:57.767573] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-26 14:53:57.807080] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-26 14:53:57.807678] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_76.output'),
                                  'state': {'begin': 1666781746.7566874,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_xnvme_io_uring.conf '
                                                   '-q 4 -o 512 -w randread -t '
                                                   "10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.076358556747437,
                                            'end': 1666781757.833046,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_76.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_76.state')},
                       'cmd_77': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_77.output'),
                                  'state': {'begin': 1666781757.8356624,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_xnvme_io_uring.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.8008663654327393,
                                            'end': 1666781758.6365287,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_77.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_77.state')},
                       'cmd_78': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :  '
                                            '363187.05 IOPS     177.34 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :  '
                                            '363187.05 IOPS     177.34 MiB/s\n'
                                            '[2022-10-26 14:54:10.638786] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-26 14:54:10.638833] [ '
                                            'DPDK EAL parameters: [2022-10-26 '
                                            '14:54:10.638840] bdevperf '
                                            '[2022-10-26 14:54:10.638845] '
                                            '--no-shconf [2022-10-26 '
                                            '14:54:10.638850] -l 0 [2022-10-26 '
                                            '14:54:10.638854] --huge-unlink '
                                            '[2022-10-26 14:54:10.638858] '
                                            '--log-level=lib.eal:6 [2022-10-26 '
                                            '14:54:10.638862] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-26 14:54:10.638866] '
                                            '--log-level=user1:6 [2022-10-26 '
                                            '14:54:10.638871] --iova-mode=pa '
                                            '[2022-10-26 14:54:10.638876] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-26 14:54:10.638880] '
                                            '--match-allocations [2022-10-26 '
                                            '14:54:10.638884] '
                                            '--file-prefix=spdk_pid4368 '
                                            '[2022-10-26 14:54:10.638888] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-26 14:54:10.760857] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-26 14:54:10.799163] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-26 14:54:10.799767] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_78.output'),
                                  'state': {'begin': 1666781759.7290328,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_xnvme_io_uring.conf '
                                                   '-q 4 -o 512 -w randread -t '
                                                   "10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.097384691238403,
                                            'end': 1666781770.8264174,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_78.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_78.state')},
                       'cmd_79': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_79.output'),
                                  'state': {'begin': 1666781770.8294969,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_aio_foo.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.8280513286590576,
                                            'end': 1666781771.6575482,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_79.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_79.state')},
                       'cmd_80': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :  '
                                            '384959.50 IOPS     187.97 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :  '
                                            '384959.50 IOPS     187.97 MiB/s\n'
                                            '[2022-10-26 14:54:23.653940] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-26 14:54:23.653988] [ '
                                            'DPDK EAL parameters: [2022-10-26 '
                                            '14:54:23.653994] bdevperf '
                                            '[2022-10-26 14:54:23.653998] '
                                            '--no-shconf [2022-10-26 '
                                            '14:54:23.654003] -l 0 [2022-10-26 '
                                            '14:54:23.654007] --huge-unlink '
                                            '[2022-10-26 14:54:23.654012] '
                                            '--log-level=lib.eal:6 [2022-10-26 '
                                            '14:54:23.654017] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-26 14:54:23.654022] '
                                            '--log-level=user1:6 [2022-10-26 '
                                            '14:54:23.654026] --iova-mode=pa '
                                            '[2022-10-26 14:54:23.654030] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-26 14:54:23.654035] '
                                            '--match-allocations [2022-10-26 '
                                            '14:54:23.654040] '
                                            '--file-prefix=spdk_pid4393 '
                                            '[2022-10-26 14:54:23.654045] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-26 14:54:23.776052] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-26 14:54:23.828570] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-26 14:54:23.829162] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_80.output'),
                                  'state': {'begin': 1666781772.7136788,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_aio_foo.conf '
                                                   '-q 4 -o 512 -w randread -t '
                                                   "10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.202033996582031,
                                            'end': 1666781783.9157128,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_80.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_80.state')},
                       'cmd_81': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_81.output'),
                                  'state': {'begin': 1666781783.9179158,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_aio_foo.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.8477661609649658,
                                            'end': 1666781784.765682,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_81.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_81.state')},
                       'cmd_82': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :  '
                                            '384611.58 IOPS     187.80 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :  '
                                            '384611.58 IOPS     187.80 MiB/s\n'
                                            '[2022-10-26 14:54:36.787399] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-26 14:54:36.787459] [ '
                                            'DPDK EAL parameters: [2022-10-26 '
                                            '14:54:36.787467] bdevperf '
                                            '[2022-10-26 14:54:36.787474] '
                                            '--no-shconf [2022-10-26 '
                                            '14:54:36.787481] -l 0 [2022-10-26 '
                                            '14:54:36.787486] --huge-unlink '
                                            '[2022-10-26 14:54:36.787491] '
                                            '--log-level=lib.eal:6 [2022-10-26 '
                                            '14:54:36.787497] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-26 14:54:36.787503] '
                                            '--log-level=user1:6 [2022-10-26 '
                                            '14:54:36.787510] --iova-mode=pa '
                                            '[2022-10-26 14:54:36.787516] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-26 14:54:36.787523] '
                                            '--match-allocations [2022-10-26 '
                                            '14:54:36.787530] '
                                            '--file-prefix=spdk_pid4418 '
                                            '[2022-10-26 14:54:36.787537] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-26 14:54:36.909401] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-26 14:54:36.951018] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-26 14:54:36.951625] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_82.output'),
                                  'state': {'begin': 1666781785.8780043,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_aio_foo.conf '
                                                   '-q 4 -o 512 -w randread -t '
                                                   "10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.161867618560791,
                                            'end': 1666781797.039872,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_82.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_82.state')},
                       'cmd_83': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_83.output'),
                                  'state': {'begin': 1666781797.0430489,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_aio_foo.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.8415968418121338,
                                            'end': 1666781797.8846457,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_83.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_83.state')},
                       'cmd_84': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :  '
                                            '387768.06 IOPS     189.34 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :  '
                                            '387768.06 IOPS     189.34 MiB/s\n'
                                            '[2022-10-26 14:54:49.793371] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-26 14:54:49.793413] [ '
                                            'DPDK EAL parameters: [2022-10-26 '
                                            '14:54:49.793420] bdevperf '
                                            '[2022-10-26 14:54:49.793424] '
                                            '--no-shconf [2022-10-26 '
                                            '14:54:49.793429] -l 0 [2022-10-26 '
                                            '14:54:49.793434] --huge-unlink '
                                            '[2022-10-26 14:54:49.793438] '
                                            '--log-level=lib.eal:6 [2022-10-26 '
                                            '14:54:49.793444] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-26 14:54:49.793448] '
                                            '--log-level=user1:6 [2022-10-26 '
                                            '14:54:49.793452] --iova-mode=pa '
                                            '[2022-10-26 14:54:49.793456] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-26 14:54:49.793460] '
                                            '--match-allocations [2022-10-26 '
                                            '14:54:49.793466] '
                                            '--file-prefix=spdk_pid4444 '
                                            '[2022-10-26 14:54:49.793470] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-26 14:54:49.915385] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-26 14:54:49.955076] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-26 14:54:49.955675] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_84.output'),
                                  'state': {'begin': 1666781798.9065425,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_aio_foo.conf '
                                                   '-q 4 -o 512 -w randread -t '
                                                   "10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.138028144836426,
                                            'end': 1666781810.0445707,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_84.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_84.state')},
                       'cmd_85': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_85.output'),
                                  'state': {'begin': 1666781810.0475254,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_xnvme_libaio.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.8003859519958496,
                                            'end': 1666781810.8479114,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_85.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_85.state')},
                       'cmd_86': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :  '
                                            '336723.59 IOPS     164.42 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :  '
                                            '336723.59 IOPS     164.42 MiB/s\n'
                                            '[2022-10-26 14:55:02.826118] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-26 14:55:02.826165] [ '
                                            'DPDK EAL parameters: [2022-10-26 '
                                            '14:55:02.826170] bdevperf '
                                            '[2022-10-26 14:55:02.826176] '
                                            '--no-shconf [2022-10-26 '
                                            '14:55:02.826180] -l 0 [2022-10-26 '
                                            '14:55:02.826185] --huge-unlink '
                                            '[2022-10-26 14:55:02.826189] '
                                            '--log-level=lib.eal:6 [2022-10-26 '
                                            '14:55:02.826194] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-26 14:55:02.826198] '
                                            '--log-level=user1:6 [2022-10-26 '
                                            '14:55:02.826202] --iova-mode=pa '
                                            '[2022-10-26 14:55:02.826206] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-26 14:55:02.826210] '
                                            '--match-allocations [2022-10-26 '
                                            '14:55:02.826215] '
                                            '--file-prefix=spdk_pid4469 '
                                            '[2022-10-26 14:55:02.826220] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-26 14:55:02.948254] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-26 14:55:02.990987] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-26 14:55:02.991582] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_86.output'),
                                  'state': {'begin': 1666781811.881793,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_xnvme_libaio.conf '
                                                   '-q 4 -o 512 -w randread -t '
                                                   "10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.215790033340454,
                                            'end': 1666781823.097583,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_86.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_86.state')},
                       'cmd_87': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_87.output'),
                                  'state': {'begin': 1666781823.1001909,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_xnvme_libaio.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.8029780387878418,
                                            'end': 1666781823.903169,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_87.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_87.state')},
                       'cmd_88': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :  '
                                            '338676.43 IOPS     165.37 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :  '
                                            '338676.43 IOPS     165.37 MiB/s\n'
                                            '[2022-10-26 14:55:15.880904] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-26 14:55:15.880951] [ '
                                            'DPDK EAL parameters: [2022-10-26 '
                                            '14:55:15.880957] bdevperf '
                                            '[2022-10-26 14:55:15.880962] '
                                            '--no-shconf [2022-10-26 '
                                            '14:55:15.880966] -l 0 [2022-10-26 '
                                            '14:55:15.880971] --huge-unlink '
                                            '[2022-10-26 14:55:15.880976] '
                                            '--log-level=lib.eal:6 [2022-10-26 '
                                            '14:55:15.880981] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-26 14:55:15.880985] '
                                            '--log-level=user1:6 [2022-10-26 '
                                            '14:55:15.880989] --iova-mode=pa '
                                            '[2022-10-26 14:55:15.880995] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-26 14:55:15.881000] '
                                            '--match-allocations [2022-10-26 '
                                            '14:55:15.881004] '
                                            '--file-prefix=spdk_pid4494 '
                                            '[2022-10-26 14:55:15.881009] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-26 14:55:16.003039] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-26 14:55:16.042971] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-26 14:55:16.043568] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_88.output'),
                                  'state': {'begin': 1666781824.9668732,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_xnvme_libaio.conf '
                                                   '-q 4 -o 512 -w randread -t '
                                                   "10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.181619882583618,
                                            'end': 1666781836.148493,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_88.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_88.state')},
                       'cmd_89': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_89.output'),
                                  'state': {'begin': 1666781836.1511254,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_xnvme_libaio.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.8683156967163086,
                                            'end': 1666781837.0194411,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_89.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_89.state')},
                       'cmd_90': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :  '
                                            '337224.20 IOPS     164.66 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :  '
                                            '337224.20 IOPS     164.66 MiB/s\n'
                                            '[2022-10-26 14:55:29.004977] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-26 14:55:29.005022] [ '
                                            'DPDK EAL parameters: [2022-10-26 '
                                            '14:55:29.005028] bdevperf '
                                            '[2022-10-26 14:55:29.005033] '
                                            '--no-shconf [2022-10-26 '
                                            '14:55:29.005037] -l 0 [2022-10-26 '
                                            '14:55:29.005042] --huge-unlink '
                                            '[2022-10-26 14:55:29.005046] '
                                            '--log-level=lib.eal:6 [2022-10-26 '
                                            '14:55:29.005051] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-26 14:55:29.005055] '
                                            '--log-level=user1:6 [2022-10-26 '
                                            '14:55:29.005060] --iova-mode=pa '
                                            '[2022-10-26 14:55:29.005064] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-26 14:55:29.005068] '
                                            '--match-allocations [2022-10-26 '
                                            '14:55:29.005072] '
                                            '--file-prefix=spdk_pid4520 '
                                            '[2022-10-26 14:55:29.005077] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-26 14:55:29.127554] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-26 14:55:29.171155] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-26 14:55:29.171762] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_90.output'),
                                  'state': {'begin': 1666781838.11049,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_xnvme_libaio.conf '
                                                   '-q 4 -o 512 -w randread -t '
                                                   "10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.155629873275757,
                                            'end': 1666781849.26612,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_90.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_90.state')},
                       'cmd_91': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_91.output'),
                                  'state': {'begin': 1666781849.2691355,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.8817453384399414,
                                            'end': 1666781850.1508808,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_91.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_91.state')},
                       'cmd_92': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :  '
                                            '559323.89 IOPS     273.11 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :  '
                                            '559323.89 IOPS     273.11 MiB/s\n'
                                            '[2022-10-26 14:55:42.157720] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-26 14:55:42.157764] [ '
                                            'DPDK EAL parameters: [2022-10-26 '
                                            '14:55:42.157769] bdevperf '
                                            '[2022-10-26 14:55:42.157774] '
                                            '--no-shconf [2022-10-26 '
                                            '14:55:42.157779] -l 0 [2022-10-26 '
                                            '14:55:42.157784] --huge-unlink '
                                            '[2022-10-26 14:55:42.157788] '
                                            '--log-level=lib.eal:6 [2022-10-26 '
                                            '14:55:42.157792] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-26 14:55:42.157795] '
                                            '--log-level=user1:6 [2022-10-26 '
                                            '14:55:42.157800] --iova-mode=pa '
                                            '[2022-10-26 14:55:42.157804] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-26 14:55:42.157809] '
                                            '--match-allocations [2022-10-26 '
                                            '14:55:42.157813] '
                                            '--file-prefix=spdk_pid4546 '
                                            '[2022-10-26 14:55:42.157818] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-26 14:55:42.279857] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-26 14:55:42.323115] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-26 14:55:42.323737] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_92.output'),
                                  'state': {'begin': 1666781851.218421,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
                                                   '-q 8 -o 512 -w randread -t '
                                                   "10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.138697147369385,
                                            'end': 1666781862.3571181,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_92.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_92.state')},
                       'cmd_93': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_93.output'),
                                  'state': {'begin': 1666781862.3601584,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 1.1332049369812012,
                                            'end': 1666781863.4933634,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_93.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_93.state')},
                       'cmd_94': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :  '
                                            '629634.72 IOPS     307.44 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :  '
                                            '629634.72 IOPS     307.44 MiB/s\n'
                                            '[2022-10-26 14:55:55.451763] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-26 14:55:55.451810] [ '
                                            'DPDK EAL parameters: [2022-10-26 '
                                            '14:55:55.451816] bdevperf '
                                            '[2022-10-26 14:55:55.451821] '
                                            '--no-shconf [2022-10-26 '
                                            '14:55:55.451826] -l 0 [2022-10-26 '
                                            '14:55:55.451831] --huge-unlink '
                                            '[2022-10-26 14:55:55.451836] '
                                            '--log-level=lib.eal:6 [2022-10-26 '
                                            '14:55:55.451841] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-26 14:55:55.451845] '
                                            '--log-level=user1:6 [2022-10-26 '
                                            '14:55:55.451849] --iova-mode=pa '
                                            '[2022-10-26 14:55:55.451854] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-26 14:55:55.451859] '
                                            '--match-allocations [2022-10-26 '
                                            '14:55:55.451863] '
                                            '--file-prefix=spdk_pid4572 '
                                            '[2022-10-26 14:55:55.451868] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-26 14:55:55.573975] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-26 14:55:55.607138] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-26 14:55:55.607755] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_94.output'),
                                  'state': {'begin': 1666781864.5490265,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
                                                   '-q 8 -o 512 -w randread -t '
                                                   "10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.118456840515137,
                                            'end': 1666781875.6674833,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_94.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_94.state')},
                       'cmd_95': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_95.output'),
                                  'state': {'begin': 1666781875.670253,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.8222548961639404,
                                            'end': 1666781876.492508,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_95.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_95.state')},
                       'cmd_96': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :  '
                                            '506563.20 IOPS     247.35 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :  '
                                            '506563.20 IOPS     247.35 MiB/s\n'
                                            '[2022-10-26 14:56:08.514571] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-26 14:56:08.514615] [ '
                                            'DPDK EAL parameters: [2022-10-26 '
                                            '14:56:08.514621] bdevperf '
                                            '[2022-10-26 14:56:08.514626] '
                                            '--no-shconf [2022-10-26 '
                                            '14:56:08.514631] -l 0 [2022-10-26 '
                                            '14:56:08.514636] --huge-unlink '
                                            '[2022-10-26 14:56:08.514641] '
                                            '--log-level=lib.eal:6 [2022-10-26 '
                                            '14:56:08.514645] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-26 14:56:08.514649] '
                                            '--log-level=user1:6 [2022-10-26 '
                                            '14:56:08.514653] --iova-mode=pa '
                                            '[2022-10-26 14:56:08.514657] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-26 14:56:08.514661] '
                                            '--match-allocations [2022-10-26 '
                                            '14:56:08.514667] '
                                            '--file-prefix=spdk_pid4598 '
                                            '[2022-10-26 14:56:08.514672] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-26 14:56:08.636693] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-26 14:56:08.679060] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-26 14:56:08.679646] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_96.output'),
                                  'state': {'begin': 1666781877.5796776,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
                                                   '-q 8 -o 512 -w randread -t '
                                                   "10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.161460638046265,
                                            'end': 1666781888.7411382,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_96.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_96.state')},
                       'cmd_97': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_97.output'),
                                  'state': {'begin': 1666781888.7438438,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_uring_foo.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.8231239318847656,
                                            'end': 1666781889.5669677,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_97.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_97.state')},
                       'cmd_98': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :  '
                                            '482815.98 IOPS     235.75 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :  '
                                            '482815.98 IOPS     235.75 MiB/s\n'
                                            '[2022-10-26 14:56:21.518478] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-26 14:56:21.518523] [ '
                                            'DPDK EAL parameters: [2022-10-26 '
                                            '14:56:21.518529] bdevperf '
                                            '[2022-10-26 14:56:21.518534] '
                                            '--no-shconf [2022-10-26 '
                                            '14:56:21.518538] -l 0 [2022-10-26 '
                                            '14:56:21.518543] --huge-unlink '
                                            '[2022-10-26 14:56:21.518547] '
                                            '--log-level=lib.eal:6 [2022-10-26 '
                                            '14:56:21.518551] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-26 14:56:21.518556] '
                                            '--log-level=user1:6 [2022-10-26 '
                                            '14:56:21.518560] --iova-mode=pa '
                                            '[2022-10-26 14:56:21.518564] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-26 14:56:21.518568] '
                                            '--match-allocations [2022-10-26 '
                                            '14:56:21.518575] '
                                            '--file-prefix=spdk_pid4623 '
                                            '[2022-10-26 14:56:21.518580] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-26 14:56:21.640578] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-26 14:56:21.679004] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-26 14:56:21.679588] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_98.output'),
                                  'state': {'begin': 1666781890.62481,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_uring_foo.conf '
                                                   '-q 8 -o 512 -w randread -t '
                                                   "10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.080789804458618,
                                            'end': 1666781901.7055998,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_98.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_98.state')},
                       'cmd_99': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_99.output'),
                                  'state': {'begin': 1666781901.7083747,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_uring_foo.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.9021365642547607,
                                            'end': 1666781902.6105113,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_99.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_99.state')}}},
 'id': '001_bdevperf',
 'name': 'bdevperf',
 'nr': 1,
 'status': {'elapsed': 783.6079878807068,
            'failed': 0,
            'passed': 1,
            'skipped': 0,
            'started': 0.0},
 'uses': 'bdevperf',
 'with': {'iodepths': [1, 2, 4, 8], 'iosizes': ['512'], 'repetitions': 3}}
$ rm /tmp/spdk_bdev_xnvme_io_uring_cmd.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring_cmd.conf -q 1 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  150921.39 IOPS      73.69 MiB/s

 =============================================================

 Total                       :  150921.39 IOPS      73.69 MiB/s
[2022-10-26 14:45:55.003669] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 14:45:55.003716] [ DPDK EAL parameters: [2022-10-26 14:45:55.003722] bdevperf [2022-10-26 14:45:55.003727] --no-shconf [2022-10-26 14:45:55.003731] -l 0 [2022-10-26 14:45:55.003736] --huge-unlink [2022-10-26 14:45:55.003740] --log-level=lib.eal:6 [2022-10-26 14:45:55.003744] --log-level=lib.cryptodev:5 [2022-10-26 14:45:55.003748] --log-level=user1:6 [2022-10-26 14:45:55.003753] --iova-mode=pa [2022-10-26 14:45:55.003759] --base-virtaddr=0x200000000000 [2022-10-26 14:45:55.003763] --match-allocations [2022-10-26 14:45:55.003768] --file-prefix=spdk_pid3400 [2022-10-26 14:45:55.003772] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 14:45:55.125466] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 14:45:55.168002] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 14:45:55.168555] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_xnvme_io_uring_cmd.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring_cmd.conf -q 1 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  171436.90 IOPS      83.71 MiB/s

 =============================================================

 Total                       :  171436.90 IOPS      83.71 MiB/s
[2022-10-26 14:46:08.004494] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 14:46:08.004535] [ DPDK EAL parameters: [2022-10-26 14:46:08.004541] bdevperf [2022-10-26 14:46:08.004546] --no-shconf [2022-10-26 14:46:08.004550] -l 0 [2022-10-26 14:46:08.004555] --huge-unlink [2022-10-26 14:46:08.004559] --log-level=lib.eal:6 [2022-10-26 14:46:08.004564] --log-level=lib.cryptodev:5 [2022-10-26 14:46:08.004568] --log-level=user1:6 [2022-10-26 14:46:08.004573] --iova-mode=pa [2022-10-26 14:46:08.004577] --base-virtaddr=0x200000000000 [2022-10-26 14:46:08.004588] --match-allocations [2022-10-26 14:46:08.004593] --file-prefix=spdk_pid3427 [2022-10-26 14:46:08.004597] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 14:46:08.126140] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 14:46:08.166933] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 14:46:08.167491] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_xnvme_io_uring_cmd.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring_cmd.conf -q 1 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  159095.18 IOPS      77.68 MiB/s

 =============================================================

 Total                       :  159095.18 IOPS      77.68 MiB/s
[2022-10-26 14:46:21.013312] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 14:46:21.013360] [ DPDK EAL parameters: [2022-10-26 14:46:21.013365] bdevperf [2022-10-26 14:46:21.013370] --no-shconf [2022-10-26 14:46:21.013374] -l 0 [2022-10-26 14:46:21.013379] --huge-unlink [2022-10-26 14:46:21.013383] --log-level=lib.eal:6 [2022-10-26 14:46:21.013387] --log-level=lib.cryptodev:5 [2022-10-26 14:46:21.013391] --log-level=user1:6 [2022-10-26 14:46:21.013396] --iova-mode=pa [2022-10-26 14:46:21.013400] --base-virtaddr=0x200000000000 [2022-10-26 14:46:21.013404] --match-allocations [2022-10-26 14:46:21.013408] --file-prefix=spdk_pid3453 [2022-10-26 14:46:21.013412] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 14:46:21.135461] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 14:46:21.187017] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 14:46:21.187578] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_uring_foo.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_uring_foo.conf -q 1 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  137262.72 IOPS      67.02 MiB/s

 =============================================================

 Total                       :  137262.72 IOPS      67.02 MiB/s
[2022-10-26 14:46:34.208092] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 14:46:34.208139] [ DPDK EAL parameters: [2022-10-26 14:46:34.208145] bdevperf [2022-10-26 14:46:34.208150] --no-shconf [2022-10-26 14:46:34.208154] -l 0 [2022-10-26 14:46:34.208158] --huge-unlink [2022-10-26 14:46:34.208164] --log-level=lib.eal:6 [2022-10-26 14:46:34.208168] --log-level=lib.cryptodev:5 [2022-10-26 14:46:34.208172] --log-level=user1:6 [2022-10-26 14:46:34.208177] --iova-mode=pa [2022-10-26 14:46:34.208181] --base-virtaddr=0x200000000000 [2022-10-26 14:46:34.208187] --match-allocations [2022-10-26 14:46:34.208191] --file-prefix=spdk_pid3478 [2022-10-26 14:46:34.208195] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 14:46:34.330276] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 14:46:34.370957] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 14:46:34.371536] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_uring_foo.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_uring_foo.conf -q 1 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  138058.37 IOPS      67.41 MiB/s

 =============================================================

 Total                       :  138058.37 IOPS      67.41 MiB/s
[2022-10-26 14:46:47.231816] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 14:46:47.231858] [ DPDK EAL parameters: [2022-10-26 14:46:47.231864] bdevperf [2022-10-26 14:46:47.231868] --no-shconf [2022-10-26 14:46:47.231872] -l 0 [2022-10-26 14:46:47.231877] --huge-unlink [2022-10-26 14:46:47.231881] --log-level=lib.eal:6 [2022-10-26 14:46:47.231885] --log-level=lib.cryptodev:5 [2022-10-26 14:46:47.231889] --log-level=user1:6 [2022-10-26 14:46:47.231893] --iova-mode=pa [2022-10-26 14:46:47.231898] --base-virtaddr=0x200000000000 [2022-10-26 14:46:47.231902] --match-allocations [2022-10-26 14:46:47.231906] --file-prefix=spdk_pid3503 [2022-10-26 14:46:47.231910] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 14:46:47.353497] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 14:46:47.394950] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 14:46:47.395495] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_uring_foo.conf -q 8 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  482862.92 IOPS     235.77 MiB/s

 =============================================================

 Total                       :  482862.92 IOPS     235.77 MiB/s
[2022-10-26 14:56:34.681870] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 14:56:34.681913] [ DPDK EAL parameters: [2022-10-26 14:56:34.681919] bdevperf [2022-10-26 14:56:34.681924] --no-shconf [2022-10-26 14:56:34.681928] -l 0 [2022-10-26 14:56:34.681932] --huge-unlink [2022-10-26 14:56:34.681937] --log-level=lib.eal:6 [2022-10-26 14:56:34.681941] --log-level=lib.cryptodev:5 [2022-10-26 14:56:34.681945] --log-level=user1:6 [2022-10-26 14:56:34.681949] --iova-mode=pa [2022-10-26 14:56:34.681952] --base-virtaddr=0x200000000000 [2022-10-26 14:56:34.681957] --match-allocations [2022-10-26 14:56:34.681961] --file-prefix=spdk_pid4648 [2022-10-26 14:56:34.681966] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 14:56:34.804031] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 14:56:34.842979] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 14:56:34.843588] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_uring_foo.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_uring_foo.conf -q 8 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  482460.34 IOPS     235.58 MiB/s

 =============================================================

 Total                       :  482460.34 IOPS     235.58 MiB/s
[2022-10-26 14:56:47.633077] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 14:56:47.633125] [ DPDK EAL parameters: [2022-10-26 14:56:47.633131] bdevperf [2022-10-26 14:56:47.633136] --no-shconf [2022-10-26 14:56:47.633141] -l 0 [2022-10-26 14:56:47.633146] --huge-unlink [2022-10-26 14:56:47.633150] --log-level=lib.eal:6 [2022-10-26 14:56:47.633155] --log-level=lib.cryptodev:5 [2022-10-26 14:56:47.633160] --log-level=user1:6 [2022-10-26 14:56:47.633164] --iova-mode=pa [2022-10-26 14:56:47.633169] --base-virtaddr=0x200000000000 [2022-10-26 14:56:47.633174] --match-allocations [2022-10-26 14:56:47.633178] --file-prefix=spdk_pid4673 [2022-10-26 14:56:47.633183] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 14:56:47.755367] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 14:56:47.798960] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 14:56:47.799547] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_xnvme_io_uring.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring.conf -q 8 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  395076.98 IOPS     192.91 MiB/s

 =============================================================

 Total                       :  395076.98 IOPS     192.91 MiB/s
[2022-10-26 14:57:00.627211] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 14:57:00.627256] [ DPDK EAL parameters: [2022-10-26 14:57:00.627262] bdevperf [2022-10-26 14:57:00.627267] --no-shconf [2022-10-26 14:57:00.627271] -l 0 [2022-10-26 14:57:00.627276] --huge-unlink [2022-10-26 14:57:00.627280] --log-level=lib.eal:6 [2022-10-26 14:57:00.627283] --log-level=lib.cryptodev:5 [2022-10-26 14:57:00.627287] --log-level=user1:6 [2022-10-26 14:57:00.627292] --iova-mode=pa [2022-10-26 14:57:00.627297] --base-virtaddr=0x200000000000 [2022-10-26 14:57:00.627301] --match-allocations [2022-10-26 14:57:00.627305] --file-prefix=spdk_pid4698 [2022-10-26 14:57:00.627309] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 14:57:00.749257] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 14:57:00.791057] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 14:57:00.791650] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_xnvme_io_uring.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring.conf -q 8 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  392195.29 IOPS     191.50 MiB/s

 =============================================================

 Total                       :  392195.29 IOPS     191.50 MiB/s
[2022-10-26 14:57:13.666086] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 14:57:13.666140] [ DPDK EAL parameters: [2022-10-26 14:57:13.666148] bdevperf [2022-10-26 14:57:13.666155] --no-shconf [2022-10-26 14:57:13.666161] -l 0 [2022-10-26 14:57:13.666168] --huge-unlink [2022-10-26 14:57:13.666173] --log-level=lib.eal:6 [2022-10-26 14:57:13.666179] --log-level=lib.cryptodev:5 [2022-10-26 14:57:13.666185] --log-level=user1:6 [2022-10-26 14:57:13.666192] --iova-mode=pa [2022-10-26 14:57:13.666199] --base-virtaddr=0x200000000000 [2022-10-26 14:57:13.666206] --match-allocations [2022-10-26 14:57:13.666213] --file-prefix=spdk_pid4723 [2022-10-26 14:57:13.666219] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 14:57:13.788213] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 14:57:13.831019] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 14:57:13.831606] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_xnvme_io_uring.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring.conf -q 8 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  390867.35 IOPS     190.85 MiB/s

 =============================================================

 Total                       :  390867.35 IOPS     190.85 MiB/s
[2022-10-26 14:57:26.619274] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 14:57:26.619322] [ DPDK EAL parameters: [2022-10-26 14:57:26.619328] bdevperf [2022-10-26 14:57:26.619332] --no-shconf [2022-10-26 14:57:26.619337] -l 0 [2022-10-26 14:57:26.619342] --huge-unlink [2022-10-26 14:57:26.619348] --log-level=lib.eal:6 [2022-10-26 14:57:26.619353] --log-level=lib.cryptodev:5 [2022-10-26 14:57:26.619358] --log-level=user1:6 [2022-10-26 14:57:26.619362] --iova-mode=pa [2022-10-26 14:57:26.619366] --base-virtaddr=0x200000000000 [2022-10-26 14:57:26.619371] --match-allocations [2022-10-26 14:57:26.619376] --file-prefix=spdk_pid4748 [2022-10-26 14:57:26.619380] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 14:57:26.741441] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 14:57:26.791036] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 14:57:26.791643] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_aio_foo.conf || true
$ rm /tmp/spdk_bdev_uring_foo.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_aio_foo.conf -q 8 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  404566.90 IOPS     197.54 MiB/s

 =============================================================

 Total                       :  404566.90 IOPS     197.54 MiB/s
[2022-10-26 14:57:39.566766] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 14:57:39.566809] [ DPDK EAL parameters: [2022-10-26 14:57:39.566815] bdevperf [2022-10-26 14:57:39.566821] --no-shconf [2022-10-26 14:57:39.566825] -l 0 [2022-10-26 14:57:39.566829] --huge-unlink [2022-10-26 14:57:39.566834] --log-level=lib.eal:6 [2022-10-26 14:57:39.566838] --log-level=lib.cryptodev:5 [2022-10-26 14:57:39.566842] --log-level=user1:6 [2022-10-26 14:57:39.566846] --iova-mode=pa [2022-10-26 14:57:39.566850] --base-virtaddr=0x200000000000 [2022-10-26 14:57:39.566855] --match-allocations [2022-10-26 14:57:39.566859] --file-prefix=spdk_pid4773 [2022-10-26 14:57:39.566863] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 14:57:39.688845] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 14:57:39.731022] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 14:57:39.731598] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_aio_foo.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_aio_foo.conf -q 8 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  404402.85 IOPS     197.46 MiB/s

 =============================================================

 Total                       :  404402.85 IOPS     197.46 MiB/s
[2022-10-26 14:57:52.662483] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 14:57:52.662525] [ DPDK EAL parameters: [2022-10-26 14:57:52.662531] bdevperf [2022-10-26 14:57:52.662536] --no-shconf [2022-10-26 14:57:52.662540] -l 0 [2022-10-26 14:57:52.662545] --huge-unlink [2022-10-26 14:57:52.662549] --log-level=lib.eal:6 [2022-10-26 14:57:52.662553] --log-level=lib.cryptodev:5 [2022-10-26 14:57:52.662557] --log-level=user1:6 [2022-10-26 14:57:52.662561] --iova-mode=pa [2022-10-26 14:57:52.662565] --base-virtaddr=0x200000000000 [2022-10-26 14:57:52.662570] --match-allocations [2022-10-26 14:57:52.662574] --file-prefix=spdk_pid4799 [2022-10-26 14:57:52.662579] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 14:57:52.784566] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 14:57:52.827141] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 14:57:52.827734] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_aio_foo.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_aio_foo.conf -q 8 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  403868.55 IOPS     197.20 MiB/s

 =============================================================

 Total                       :  403868.55 IOPS     197.20 MiB/s
[2022-10-26 14:58:05.759802] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 14:58:05.759849] [ DPDK EAL parameters: [2022-10-26 14:58:05.759855] bdevperf [2022-10-26 14:58:05.759860] --no-shconf [2022-10-26 14:58:05.759864] -l 0 [2022-10-26 14:58:05.759869] --huge-unlink [2022-10-26 14:58:05.759874] --log-level=lib.eal:6 [2022-10-26 14:58:05.759878] --log-level=lib.cryptodev:5 [2022-10-26 14:58:05.759883] --log-level=user1:6 [2022-10-26 14:58:05.759887] --iova-mode=pa [2022-10-26 14:58:05.759891] --base-virtaddr=0x200000000000 [2022-10-26 14:58:05.759895] --match-allocations [2022-10-26 14:58:05.759900] --file-prefix=spdk_pid4825 [2022-10-26 14:58:05.759904] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 14:58:05.881869] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 14:58:05.919038] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 14:58:05.919619] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_xnvme_libaio.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_libaio.conf -q 8 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  382108.42 IOPS     186.58 MiB/s

 =============================================================

 Total                       :  382108.42 IOPS     186.58 MiB/s
[2022-10-26 14:58:18.828990] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 14:58:18.829036] [ DPDK EAL parameters: [2022-10-26 14:58:18.829042] bdevperf [2022-10-26 14:58:18.829047] --no-shconf [2022-10-26 14:58:18.829051] -l 0 [2022-10-26 14:58:18.829056] --huge-unlink [2022-10-26 14:58:18.829061] --log-level=lib.eal:6 [2022-10-26 14:58:18.829065] --log-level=lib.cryptodev:5 [2022-10-26 14:58:18.829070] --log-level=user1:6 [2022-10-26 14:58:18.829074] --iova-mode=pa [2022-10-26 14:58:18.829078] --base-virtaddr=0x200000000000 [2022-10-26 14:58:18.829083] --match-allocations [2022-10-26 14:58:18.829088] --file-prefix=spdk_pid4850 [2022-10-26 14:58:18.829092] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 14:58:18.951199] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 14:58:18.995101] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 14:58:18.995684] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_xnvme_libaio.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_libaio.conf -q 8 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  380577.08 IOPS     185.83 MiB/s

 =============================================================

 Total                       :  380577.08 IOPS     185.83 MiB/s
[2022-10-26 14:58:32.287850] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 14:58:32.287904] [ DPDK EAL parameters: [2022-10-26 14:58:32.287910] bdevperf [2022-10-26 14:58:32.287915] --no-shconf [2022-10-26 14:58:32.287920] -l 0 [2022-10-26 14:58:32.287924] --huge-unlink [2022-10-26 14:58:32.287929] --log-level=lib.eal:6 [2022-10-26 14:58:32.287933] --log-level=lib.cryptodev:5 [2022-10-26 14:58:32.287937] --log-level=user1:6 [2022-10-26 14:58:32.287942] --iova-mode=pa [2022-10-26 14:58:32.287946] --base-virtaddr=0x200000000000 [2022-10-26 14:58:32.287950] --match-allocations [2022-10-26 14:58:32.287955] --file-prefix=spdk_pid4875 [2022-10-26 14:58:32.287960] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 14:58:32.409866] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 14:58:32.450980] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 14:58:32.451586] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_xnvme_libaio.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_uring_foo.conf -q 1 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  137312.00 IOPS      67.05 MiB/s

 =============================================================

 Total                       :  137312.00 IOPS      67.05 MiB/s
[2022-10-26 14:47:00.239018] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 14:47:00.239062] [ DPDK EAL parameters: [2022-10-26 14:47:00.239067] bdevperf [2022-10-26 14:47:00.239073] --no-shconf [2022-10-26 14:47:00.239077] -l 0 [2022-10-26 14:47:00.239082] --huge-unlink [2022-10-26 14:47:00.239086] --log-level=lib.eal:6 [2022-10-26 14:47:00.239091] --log-level=lib.cryptodev:5 [2022-10-26 14:47:00.239096] --log-level=user1:6 [2022-10-26 14:47:00.239100] --iova-mode=pa [2022-10-26 14:47:00.239104] --base-virtaddr=0x200000000000 [2022-10-26 14:47:00.239108] --match-allocations [2022-10-26 14:47:00.239112] --file-prefix=spdk_pid3528 [2022-10-26 14:47:00.239117] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 14:47:00.360863] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 14:47:00.406981] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 14:47:00.407550] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_libaio.conf -q 8 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  379284.08 IOPS     185.20 MiB/s

 =============================================================

 Total                       :  379284.08 IOPS     185.20 MiB/s
[2022-10-26 14:58:45.398827] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 14:58:45.398870] [ DPDK EAL parameters: [2022-10-26 14:58:45.398876] bdevperf [2022-10-26 14:58:45.398881] --no-shconf [2022-10-26 14:58:45.398885] -l 0 [2022-10-26 14:58:45.398890] --huge-unlink [2022-10-26 14:58:45.398895] --log-level=lib.eal:6 [2022-10-26 14:58:45.398899] --log-level=lib.cryptodev:5 [2022-10-26 14:58:45.398904] --log-level=user1:6 [2022-10-26 14:58:45.398908] --iova-mode=pa [2022-10-26 14:58:45.398912] --base-virtaddr=0x200000000000 [2022-10-26 14:58:45.398918] --match-allocations [2022-10-26 14:58:45.398924] --file-prefix=spdk_pid4900 [2022-10-26 14:58:45.398929] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 14:58:45.521201] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 14:58:45.571063] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 14:58:45.571640] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_xnvme_io_uring.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring.conf -q 1 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  156589.41 IOPS      76.46 MiB/s

 =============================================================

 Total                       :  156589.41 IOPS      76.46 MiB/s
[2022-10-26 14:47:13.264251] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 14:47:13.264293] [ DPDK EAL parameters: [2022-10-26 14:47:13.264298] bdevperf [2022-10-26 14:47:13.264303] --no-shconf [2022-10-26 14:47:13.264307] -l 0 [2022-10-26 14:47:13.264311] --huge-unlink [2022-10-26 14:47:13.264316] --log-level=lib.eal:6 [2022-10-26 14:47:13.264320] --log-level=lib.cryptodev:5 [2022-10-26 14:47:13.264324] --log-level=user1:6 [2022-10-26 14:47:13.264328] --iova-mode=pa [2022-10-26 14:47:13.264332] --base-virtaddr=0x200000000000 [2022-10-26 14:47:13.264336] --match-allocations [2022-10-26 14:47:13.264340] --file-prefix=spdk_pid3553 [2022-10-26 14:47:13.264344] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 14:47:13.386330] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 14:47:13.430961] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 14:47:13.431535] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_xnvme_io_uring.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring.conf -q 1 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  156437.11 IOPS      76.39 MiB/s

 =============================================================

 Total                       :  156437.11 IOPS      76.39 MiB/s
[2022-10-26 14:47:26.307921] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 14:47:26.307971] [ DPDK EAL parameters: [2022-10-26 14:47:26.307978] bdevperf [2022-10-26 14:47:26.307985] --no-shconf [2022-10-26 14:47:26.307992] -l 0 [2022-10-26 14:47:26.307997] --huge-unlink [2022-10-26 14:47:26.308003] --log-level=lib.eal:6 [2022-10-26 14:47:26.308009] --log-level=lib.cryptodev:5 [2022-10-26 14:47:26.308016] --log-level=user1:6 [2022-10-26 14:47:26.308022] --iova-mode=pa [2022-10-26 14:47:26.308028] --base-virtaddr=0x200000000000 [2022-10-26 14:47:26.308034] --match-allocations [2022-10-26 14:47:26.308041] --file-prefix=spdk_pid3578 [2022-10-26 14:47:26.308053] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 14:47:26.429664] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 14:47:26.462979] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 14:47:26.463535] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_xnvme_io_uring.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring.conf -q 1 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  156357.41 IOPS      76.35 MiB/s

 =============================================================

 Total                       :  156357.41 IOPS      76.35 MiB/s
[2022-10-26 14:47:39.245614] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 14:47:39.245656] [ DPDK EAL parameters: [2022-10-26 14:47:39.245662] bdevperf [2022-10-26 14:47:39.245667] --no-shconf [2022-10-26 14:47:39.245672] -l 0 [2022-10-26 14:47:39.245676] --huge-unlink [2022-10-26 14:47:39.245681] --log-level=lib.eal:6 [2022-10-26 14:47:39.245685] --log-level=lib.cryptodev:5 [2022-10-26 14:47:39.245689] --log-level=user1:6 [2022-10-26 14:47:39.245694] --iova-mode=pa [2022-10-26 14:47:39.245698] --base-virtaddr=0x200000000000 [2022-10-26 14:47:39.245703] --match-allocations [2022-10-26 14:47:39.245708] --file-prefix=spdk_pid3603 [2022-10-26 14:47:39.245712] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 14:47:39.367532] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 14:47:39.411007] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 14:47:39.411561] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_aio_foo.conf || true
rm: cannot remove '/tmp/spdk_bdev_aio_foo.conf': No such file or directory

$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_aio_foo.conf -q 1 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  154952.02 IOPS      75.66 MiB/s

 =============================================================

 Total                       :  154952.02 IOPS      75.66 MiB/s
[2022-10-26 14:47:52.225176] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 14:47:52.225221] [ DPDK EAL parameters: [2022-10-26 14:47:52.225227] bdevperf [2022-10-26 14:47:52.225232] --no-shconf [2022-10-26 14:47:52.225236] -l 0 [2022-10-26 14:47:52.225240] --huge-unlink [2022-10-26 14:47:52.225245] --log-level=lib.eal:6 [2022-10-26 14:47:52.225249] --log-level=lib.cryptodev:5 [2022-10-26 14:47:52.225253] --log-level=user1:6 [2022-10-26 14:47:52.225257] --iova-mode=pa [2022-10-26 14:47:52.225261] --base-virtaddr=0x200000000000 [2022-10-26 14:47:52.225265] --match-allocations [2022-10-26 14:47:52.225270] --file-prefix=spdk_pid3628 [2022-10-26 14:47:52.225276] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 14:47:52.346983] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 14:47:52.398946] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 14:47:52.399501] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_aio_foo.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_aio_foo.conf -q 1 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  154636.84 IOPS      75.51 MiB/s

 =============================================================

 Total                       :  154636.84 IOPS      75.51 MiB/s
[2022-10-26 14:48:05.239169] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 14:48:05.239212] [ DPDK EAL parameters: [2022-10-26 14:48:05.239218] bdevperf [2022-10-26 14:48:05.239223] --no-shconf [2022-10-26 14:48:05.239228] -l 0 [2022-10-26 14:48:05.239233] --huge-unlink [2022-10-26 14:48:05.239237] --log-level=lib.eal:6 [2022-10-26 14:48:05.239241] --log-level=lib.cryptodev:5 [2022-10-26 14:48:05.239246] --log-level=user1:6 [2022-10-26 14:48:05.239250] --iova-mode=pa [2022-10-26 14:48:05.239254] --base-virtaddr=0x200000000000 [2022-10-26 14:48:05.239258] --match-allocations [2022-10-26 14:48:05.239264] --file-prefix=spdk_pid3653 [2022-10-26 14:48:05.239268] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 14:48:05.361024] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 14:48:05.410953] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 14:48:05.411512] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_aio_foo.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_aio_foo.conf -q 1 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  154684.51 IOPS      75.53 MiB/s

 =============================================================

 Total                       :  154684.51 IOPS      75.53 MiB/s
[2022-10-26 14:48:18.287627] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 14:48:18.287671] [ DPDK EAL parameters: [2022-10-26 14:48:18.287677] bdevperf [2022-10-26 14:48:18.287682] --no-shconf [2022-10-26 14:48:18.287686] -l 0 [2022-10-26 14:48:18.287691] --huge-unlink [2022-10-26 14:48:18.287695] --log-level=lib.eal:6 [2022-10-26 14:48:18.287700] --log-level=lib.cryptodev:5 [2022-10-26 14:48:18.287703] --log-level=user1:6 [2022-10-26 14:48:18.287708] --iova-mode=pa [2022-10-26 14:48:18.287712] --base-virtaddr=0x200000000000 [2022-10-26 14:48:18.287716] --match-allocations [2022-10-26 14:48:18.287721] --file-prefix=spdk_pid3679 [2022-10-26 14:48:18.287727] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 14:48:18.409256] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 14:48:18.452516] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 14:48:18.453072] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_xnvme_libaio.conf || true
rm: cannot remove '/tmp/spdk_bdev_xnvme_libaio.conf': No such file or directory

$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_libaio.conf -q 1 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  145398.43 IOPS      71.00 MiB/s

 =============================================================

 Total                       :  145398.43 IOPS      71.00 MiB/s
[2022-10-26 14:48:31.500046] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 14:48:31.500091] [ DPDK EAL parameters: [2022-10-26 14:48:31.500096] bdevperf [2022-10-26 14:48:31.500102] --no-shconf [2022-10-26 14:48:31.500106] -l 0 [2022-10-26 14:48:31.500111] --huge-unlink [2022-10-26 14:48:31.500115] --log-level=lib.eal:6 [2022-10-26 14:48:31.500119] --log-level=lib.cryptodev:5 [2022-10-26 14:48:31.500123] --log-level=user1:6 [2022-10-26 14:48:31.500128] --iova-mode=pa [2022-10-26 14:48:31.500132] --base-virtaddr=0x200000000000 [2022-10-26 14:48:31.500136] --match-allocations [2022-10-26 14:48:31.500140] --file-prefix=spdk_pid3704 [2022-10-26 14:48:31.500144] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 14:48:31.621779] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 14:48:31.662966] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 14:48:31.663515] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_xnvme_libaio.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_libaio.conf -q 1 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  146366.04 IOPS      71.47 MiB/s

 =============================================================

 Total                       :  146366.04 IOPS      71.47 MiB/s
[2022-10-26 14:48:44.543718] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 14:48:44.543768] [ DPDK EAL parameters: [2022-10-26 14:48:44.543775] bdevperf [2022-10-26 14:48:44.543780] --no-shconf [2022-10-26 14:48:44.543787] -l 0 [2022-10-26 14:48:44.543793] --huge-unlink [2022-10-26 14:48:44.543799] --log-level=lib.eal:6 [2022-10-26 14:48:44.543805] --log-level=lib.cryptodev:5 [2022-10-26 14:48:44.543810] --log-level=user1:6 [2022-10-26 14:48:44.543816] --iova-mode=pa [2022-10-26 14:48:44.543822] --base-virtaddr=0x200000000000 [2022-10-26 14:48:44.543828] --match-allocations [2022-10-26 14:48:44.543834] --file-prefix=spdk_pid3729 [2022-10-26 14:48:44.543839] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 14:48:44.665842] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 14:48:44.707021] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 14:48:44.707577] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_xnvme_libaio.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_libaio.conf -q 1 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  146618.86 IOPS      71.59 MiB/s

 =============================================================

 Total                       :  146618.86 IOPS      71.59 MiB/s
[2022-10-26 14:48:57.629719] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 14:48:57.629762] [ DPDK EAL parameters: [2022-10-26 14:48:57.629768] bdevperf [2022-10-26 14:48:57.629773] --no-shconf [2022-10-26 14:48:57.629778] -l 0 [2022-10-26 14:48:57.629782] --huge-unlink [2022-10-26 14:48:57.629785] --log-level=lib.eal:6 [2022-10-26 14:48:57.629790] --log-level=lib.cryptodev:5 [2022-10-26 14:48:57.629793] --log-level=user1:6 [2022-10-26 14:48:57.629798] --iova-mode=pa [2022-10-26 14:48:57.629801] --base-virtaddr=0x200000000000 [2022-10-26 14:48:57.629805] --match-allocations [2022-10-26 14:48:57.629809] --file-prefix=spdk_pid3755 [2022-10-26 14:48:57.629813] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 14:48:57.752067] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 14:48:57.794987] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 14:48:57.795541] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_xnvme_io_uring_cmd.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring_cmd.conf -q 2 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  261816.82 IOPS     127.84 MiB/s

 =============================================================

 Total                       :  261816.82 IOPS     127.84 MiB/s
[2022-10-26 14:49:10.728753] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 14:49:10.728804] [ DPDK EAL parameters: [2022-10-26 14:49:10.728810] bdevperf [2022-10-26 14:49:10.728814] --no-shconf [2022-10-26 14:49:10.728818] -l 0 [2022-10-26 14:49:10.728823] --huge-unlink [2022-10-26 14:49:10.728827] --log-level=lib.eal:6 [2022-10-26 14:49:10.728831] --log-level=lib.cryptodev:5 [2022-10-26 14:49:10.728835] --log-level=user1:6 [2022-10-26 14:49:10.728839] --iova-mode=pa [2022-10-26 14:49:10.728843] --base-virtaddr=0x200000000000 [2022-10-26 14:49:10.728847] --match-allocations [2022-10-26 14:49:10.728851] --file-prefix=spdk_pid3780 [2022-10-26 14:49:10.728857] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 14:49:10.850519] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 14:49:10.894989] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 14:49:10.895537] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_xnvme_io_uring_cmd.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring_cmd.conf -q 2 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  261099.59 IOPS     127.49 MiB/s

 =============================================================

 Total                       :  261099.59 IOPS     127.49 MiB/s
[2022-10-26 14:49:23.852529] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 14:49:23.852573] [ DPDK EAL parameters: [2022-10-26 14:49:23.852579] bdevperf [2022-10-26 14:49:23.852583] --no-shconf [2022-10-26 14:49:23.852589] -l 0 [2022-10-26 14:49:23.852594] --huge-unlink [2022-10-26 14:49:23.852599] --log-level=lib.eal:6 [2022-10-26 14:49:23.852603] --log-level=lib.cryptodev:5 [2022-10-26 14:49:23.852608] --log-level=user1:6 [2022-10-26 14:49:23.852612] --iova-mode=pa [2022-10-26 14:49:23.852616] --base-virtaddr=0x200000000000 [2022-10-26 14:49:23.852621] --match-allocations [2022-10-26 14:49:23.852626] --file-prefix=spdk_pid3805 [2022-10-26 14:49:23.852632] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 14:49:23.974359] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 14:49:24.027041] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 14:49:24.027602] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_xnvme_io_uring_cmd.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring_cmd.conf -q 2 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  260889.10 IOPS     127.39 MiB/s

 =============================================================

 Total                       :  260889.10 IOPS     127.39 MiB/s
[2022-10-26 14:49:36.826143] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 14:49:36.826185] [ DPDK EAL parameters: [2022-10-26 14:49:36.826191] bdevperf [2022-10-26 14:49:36.826195] --no-shconf [2022-10-26 14:49:36.826200] -l 0 [2022-10-26 14:49:36.826204] --huge-unlink [2022-10-26 14:49:36.826209] --log-level=lib.eal:6 [2022-10-26 14:49:36.826213] --log-level=lib.cryptodev:5 [2022-10-26 14:49:36.826218] --log-level=user1:6 [2022-10-26 14:49:36.826222] --iova-mode=pa [2022-10-26 14:49:36.826226] --base-virtaddr=0x200000000000 [2022-10-26 14:49:36.826230] --match-allocations [2022-10-26 14:49:36.826235] --file-prefix=spdk_pid3830 [2022-10-26 14:49:36.826239] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 14:49:36.947892] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 14:49:36.991044] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 14:49:36.991602] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_uring_foo.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_uring_foo.conf -q 2 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  230826.32 IOPS     112.71 MiB/s

 =============================================================

 Total                       :  230826.32 IOPS     112.71 MiB/s
[2022-10-26 14:49:49.866229] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 14:49:49.866273] [ DPDK EAL parameters: [2022-10-26 14:49:49.866279] bdevperf [2022-10-26 14:49:49.866283] --no-shconf [2022-10-26 14:49:49.866288] -l 0 [2022-10-26 14:49:49.866293] --huge-unlink [2022-10-26 14:49:49.866298] --log-level=lib.eal:6 [2022-10-26 14:49:49.866303] --log-level=lib.cryptodev:5 [2022-10-26 14:49:49.866307] --log-level=user1:6 [2022-10-26 14:49:49.866311] --iova-mode=pa [2022-10-26 14:49:49.866316] --base-virtaddr=0x200000000000 [2022-10-26 14:49:49.866321] --match-allocations [2022-10-26 14:49:49.866325] --file-prefix=spdk_pid3855 [2022-10-26 14:49:49.866330] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 14:49:49.987927] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 14:49:50.031010] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 14:49:50.031564] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_uring_foo.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_uring_foo.conf -q 2 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  229739.44 IOPS     112.18 MiB/s

 =============================================================

 Total                       :  229739.44 IOPS     112.18 MiB/s
[2022-10-26 14:50:02.780961] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 14:50:02.781004] [ DPDK EAL parameters: [2022-10-26 14:50:02.781010] bdevperf [2022-10-26 14:50:02.781015] --no-shconf [2022-10-26 14:50:02.781020] -l 0 [2022-10-26 14:50:02.781026] --huge-unlink [2022-10-26 14:50:02.781030] --log-level=lib.eal:6 [2022-10-26 14:50:02.781034] --log-level=lib.cryptodev:5 [2022-10-26 14:50:02.781038] --log-level=user1:6 [2022-10-26 14:50:02.781042] --iova-mode=pa [2022-10-26 14:50:02.781046] --base-virtaddr=0x200000000000 [2022-10-26 14:50:02.781050] --match-allocations [2022-10-26 14:50:02.781060] --file-prefix=spdk_pid3880 [2022-10-26 14:50:02.781064] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 14:50:02.902700] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 14:50:02.946942] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 14:50:02.947497] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_uring_foo.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_uring_foo.conf -q 2 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  231226.96 IOPS     112.90 MiB/s

 =============================================================

 Total                       :  231226.96 IOPS     112.90 MiB/s
[2022-10-26 14:50:15.813398] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 14:50:15.813439] [ DPDK EAL parameters: [2022-10-26 14:50:15.813445] bdevperf [2022-10-26 14:50:15.813450] --no-shconf [2022-10-26 14:50:15.813455] -l 0 [2022-10-26 14:50:15.813460] --huge-unlink [2022-10-26 14:50:15.813464] --log-level=lib.eal:6 [2022-10-26 14:50:15.813470] --log-level=lib.cryptodev:5 [2022-10-26 14:50:15.813474] --log-level=user1:6 [2022-10-26 14:50:15.813478] --iova-mode=pa [2022-10-26 14:50:15.813482] --base-virtaddr=0x200000000000 [2022-10-26 14:50:15.813487] --match-allocations [2022-10-26 14:50:15.813491] --file-prefix=spdk_pid3906 [2022-10-26 14:50:15.813495] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 14:50:15.935094] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 14:50:15.978913] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 14:50:15.979453] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_xnvme_io_uring.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring.conf -q 2 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  282909.29 IOPS     138.14 MiB/s

 =============================================================

 Total                       :  282909.29 IOPS     138.14 MiB/s
[2022-10-26 14:50:28.780387] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 14:50:28.780434] [ DPDK EAL parameters: [2022-10-26 14:50:28.780440] bdevperf [2022-10-26 14:50:28.780445] --no-shconf [2022-10-26 14:50:28.780449] -l 0 [2022-10-26 14:50:28.780453] --huge-unlink [2022-10-26 14:50:28.780458] --log-level=lib.eal:6 [2022-10-26 14:50:28.780463] --log-level=lib.cryptodev:5 [2022-10-26 14:50:28.780467] --log-level=user1:6 [2022-10-26 14:50:28.780471] --iova-mode=pa [2022-10-26 14:50:28.780475] --base-virtaddr=0x200000000000 [2022-10-26 14:50:28.780479] --match-allocations [2022-10-26 14:50:28.780485] --file-prefix=spdk_pid3931 [2022-10-26 14:50:28.780491] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 14:50:28.902341] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 14:50:28.948483] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 14:50:28.949036] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_xnvme_io_uring.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring.conf -q 2 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  283679.60 IOPS     138.52 MiB/s

 =============================================================

 Total                       :  283679.60 IOPS     138.52 MiB/s
[2022-10-26 14:50:41.753061] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 14:50:41.753108] [ DPDK EAL parameters: [2022-10-26 14:50:41.753113] bdevperf [2022-10-26 14:50:41.753119] --no-shconf [2022-10-26 14:50:41.753123] -l 0 [2022-10-26 14:50:41.753128] --huge-unlink [2022-10-26 14:50:41.753132] --log-level=lib.eal:6 [2022-10-26 14:50:41.753138] --log-level=lib.cryptodev:5 [2022-10-26 14:50:41.753142] --log-level=user1:6 [2022-10-26 14:50:41.753146] --iova-mode=pa [2022-10-26 14:50:41.753150] --base-virtaddr=0x200000000000 [2022-10-26 14:50:41.753154] --match-allocations [2022-10-26 14:50:41.753160] --file-prefix=spdk_pid3957 [2022-10-26 14:50:41.753164] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 14:50:41.874953] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 14:50:41.915040] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 14:50:41.915590] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_xnvme_io_uring.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring.conf -q 2 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  283155.55 IOPS     138.26 MiB/s

 =============================================================

 Total                       :  283155.55 IOPS     138.26 MiB/s
[2022-10-26 14:50:54.724079] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 14:50:54.724121] [ DPDK EAL parameters: [2022-10-26 14:50:54.724127] bdevperf [2022-10-26 14:50:54.724132] --no-shconf [2022-10-26 14:50:54.724137] -l 0 [2022-10-26 14:50:54.724141] --huge-unlink [2022-10-26 14:50:54.724146] --log-level=lib.eal:6 [2022-10-26 14:50:54.724149] --log-level=lib.cryptodev:5 [2022-10-26 14:50:54.724154] --log-level=user1:6 [2022-10-26 14:50:54.724157] --iova-mode=pa [2022-10-26 14:50:54.724161] --base-virtaddr=0x200000000000 [2022-10-26 14:50:54.724165] --match-allocations [2022-10-26 14:50:54.724169] --file-prefix=spdk_pid3984 [2022-10-26 14:50:54.724175] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 14:50:54.845795] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 14:50:54.887066] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 14:50:54.887623] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_aio_foo.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_aio_foo.conf -q 2 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  281466.49 IOPS     137.43 MiB/s

 =============================================================

 Total                       :  281466.49 IOPS     137.43 MiB/s
[2022-10-26 14:51:07.680050] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 14:51:07.680094] [ DPDK EAL parameters: [2022-10-26 14:51:07.680101] bdevperf [2022-10-26 14:51:07.680106] --no-shconf [2022-10-26 14:51:07.680111] -l 0 [2022-10-26 14:51:07.680118] --huge-unlink [2022-10-26 14:51:07.680126] --log-level=lib.eal:6 [2022-10-26 14:51:07.680132] --log-level=lib.cryptodev:5 [2022-10-26 14:51:07.680137] --log-level=user1:6 [2022-10-26 14:51:07.680142] --iova-mode=pa [2022-10-26 14:51:07.680149] --base-virtaddr=0x200000000000 [2022-10-26 14:51:07.680154] --match-allocations [2022-10-26 14:51:07.680160] --file-prefix=spdk_pid4010 [2022-10-26 14:51:07.680167] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 14:51:07.801780] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 14:51:07.839036] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 14:51:07.839594] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_aio_foo.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_aio_foo.conf -q 2 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  280747.16 IOPS     137.08 MiB/s

 =============================================================

 Total                       :  280747.16 IOPS     137.08 MiB/s
[2022-10-26 14:51:20.723914] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 14:51:20.723958] [ DPDK EAL parameters: [2022-10-26 14:51:20.723964] bdevperf [2022-10-26 14:51:20.723968] --no-shconf [2022-10-26 14:51:20.723973] -l 0 [2022-10-26 14:51:20.723979] --huge-unlink [2022-10-26 14:51:20.723983] --log-level=lib.eal:6 [2022-10-26 14:51:20.723988] --log-level=lib.cryptodev:5 [2022-10-26 14:51:20.723992] --log-level=user1:6 [2022-10-26 14:51:20.723997] --iova-mode=pa [2022-10-26 14:51:20.724001] --base-virtaddr=0x200000000000 [2022-10-26 14:51:20.724004] --match-allocations [2022-10-26 14:51:20.724009] --file-prefix=spdk_pid4036 [2022-10-26 14:51:20.724014] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 14:51:20.845500] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 14:51:20.895019] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 14:51:20.895578] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_aio_foo.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_aio_foo.conf -q 2 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  280777.96 IOPS     137.10 MiB/s

 =============================================================

 Total                       :  280777.96 IOPS     137.10 MiB/s
[2022-10-26 14:51:33.686533] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 14:51:33.686578] [ DPDK EAL parameters: [2022-10-26 14:51:33.686584] bdevperf [2022-10-26 14:51:33.686589] --no-shconf [2022-10-26 14:51:33.686594] -l 0 [2022-10-26 14:51:33.686598] --huge-unlink [2022-10-26 14:51:33.686603] --log-level=lib.eal:6 [2022-10-26 14:51:33.686608] --log-level=lib.cryptodev:5 [2022-10-26 14:51:33.686612] --log-level=user1:6 [2022-10-26 14:51:33.686616] --iova-mode=pa [2022-10-26 14:51:33.686620] --base-virtaddr=0x200000000000 [2022-10-26 14:51:33.686625] --match-allocations [2022-10-26 14:51:33.686630] --file-prefix=spdk_pid4063 [2022-10-26 14:51:33.686635] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 14:51:33.808399] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 14:51:33.851003] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 14:51:33.851560] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_xnvme_libaio.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_libaio.conf -q 2 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  267085.41 IOPS     130.41 MiB/s

 =============================================================

 Total                       :  267085.41 IOPS     130.41 MiB/s
[2022-10-26 14:51:46.743969] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 14:51:46.744011] [ DPDK EAL parameters: [2022-10-26 14:51:46.744017] bdevperf [2022-10-26 14:51:46.744021] --no-shconf [2022-10-26 14:51:46.744026] -l 0 [2022-10-26 14:51:46.744030] --huge-unlink [2022-10-26 14:51:46.744034] --log-level=lib.eal:6 [2022-10-26 14:51:46.744038] --log-level=lib.cryptodev:5 [2022-10-26 14:51:46.744044] --log-level=user1:6 [2022-10-26 14:51:46.744048] --iova-mode=pa [2022-10-26 14:51:46.744053] --base-virtaddr=0x200000000000 [2022-10-26 14:51:46.744058] --match-allocations [2022-10-26 14:51:46.744064] --file-prefix=spdk_pid4090 [2022-10-26 14:51:46.744068] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 14:51:46.866027] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 14:51:46.903064] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 14:51:46.903629] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_xnvme_libaio.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_libaio.conf -q 2 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  267177.11 IOPS     130.46 MiB/s

 =============================================================

 Total                       :  267177.11 IOPS     130.46 MiB/s
[2022-10-26 14:51:59.778212] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 14:51:59.778256] [ DPDK EAL parameters: [2022-10-26 14:51:59.778262] bdevperf [2022-10-26 14:51:59.778267] --no-shconf [2022-10-26 14:51:59.778271] -l 0 [2022-10-26 14:51:59.778276] --huge-unlink [2022-10-26 14:51:59.778281] --log-level=lib.eal:6 [2022-10-26 14:51:59.778285] --log-level=lib.cryptodev:5 [2022-10-26 14:51:59.778291] --log-level=user1:6 [2022-10-26 14:51:59.778295] --iova-mode=pa [2022-10-26 14:51:59.778299] --base-virtaddr=0x200000000000 [2022-10-26 14:51:59.778303] --match-allocations [2022-10-26 14:51:59.778307] --file-prefix=spdk_pid4115 [2022-10-26 14:51:59.778313] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 14:51:59.900002] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 14:51:59.951001] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 14:51:59.951562] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_xnvme_libaio.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_libaio.conf -q 2 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  266611.68 IOPS     130.18 MiB/s

 =============================================================

 Total                       :  266611.68 IOPS     130.18 MiB/s
[2022-10-26 14:52:12.820906] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 14:52:12.820957] [ DPDK EAL parameters: [2022-10-26 14:52:12.820965] bdevperf [2022-10-26 14:52:12.820972] --no-shconf [2022-10-26 14:52:12.820978] -l 0 [2022-10-26 14:52:12.820984] --huge-unlink [2022-10-26 14:52:12.820990] --log-level=lib.eal:6 [2022-10-26 14:52:12.820997] --log-level=lib.cryptodev:5 [2022-10-26 14:52:12.821003] --log-level=user1:6 [2022-10-26 14:52:12.821011] --iova-mode=pa [2022-10-26 14:52:12.821019] --base-virtaddr=0x200000000000 [2022-10-26 14:52:12.821025] --match-allocations [2022-10-26 14:52:12.821032] --file-prefix=spdk_pid4140 [2022-10-26 14:52:12.821039] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 14:52:12.942842] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 14:52:12.987011] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 14:52:12.987571] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_xnvme_io_uring_cmd.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring_cmd.conf -q 4 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  409829.09 IOPS     200.11 MiB/s

 =============================================================

 Total                       :  409829.09 IOPS     200.11 MiB/s
[2022-10-26 14:52:25.997001] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 14:52:25.997050] [ DPDK EAL parameters: [2022-10-26 14:52:25.997056] bdevperf [2022-10-26 14:52:25.997060] --no-shconf [2022-10-26 14:52:25.997064] -l 0 [2022-10-26 14:52:25.997069] --huge-unlink [2022-10-26 14:52:25.997074] --log-level=lib.eal:6 [2022-10-26 14:52:25.997078] --log-level=lib.cryptodev:5 [2022-10-26 14:52:25.997083] --log-level=user1:6 [2022-10-26 14:52:25.997087] --iova-mode=pa [2022-10-26 14:52:25.997091] --base-virtaddr=0x200000000000 [2022-10-26 14:52:25.997095] --match-allocations [2022-10-26 14:52:25.997100] --file-prefix=spdk_pid4165 [2022-10-26 14:52:25.997105] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 14:52:26.118822] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 14:52:26.162992] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 14:52:26.163553] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_xnvme_io_uring_cmd.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring_cmd.conf -q 4 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  453036.84 IOPS     221.21 MiB/s

 =============================================================

 Total                       :  453036.84 IOPS     221.21 MiB/s
[2022-10-26 14:52:39.039669] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 14:52:39.039712] [ DPDK EAL parameters: [2022-10-26 14:52:39.039717] bdevperf [2022-10-26 14:52:39.039722] --no-shconf [2022-10-26 14:52:39.039726] -l 0 [2022-10-26 14:52:39.039731] --huge-unlink [2022-10-26 14:52:39.039736] --log-level=lib.eal:6 [2022-10-26 14:52:39.039740] --log-level=lib.cryptodev:5 [2022-10-26 14:52:39.039745] --log-level=user1:6 [2022-10-26 14:52:39.039749] --iova-mode=pa [2022-10-26 14:52:39.039752] --base-virtaddr=0x200000000000 [2022-10-26 14:52:39.039757] --match-allocations [2022-10-26 14:52:39.039761] --file-prefix=spdk_pid4190 [2022-10-26 14:52:39.039766] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 14:52:39.161430] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 14:52:39.203055] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 14:52:39.203611] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_xnvme_io_uring_cmd.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring_cmd.conf -q 4 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  493618.05 IOPS     241.02 MiB/s

 =============================================================

 Total                       :  493618.05 IOPS     241.02 MiB/s
[2022-10-26 14:52:52.548321] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 14:52:52.548367] [ DPDK EAL parameters: [2022-10-26 14:52:52.548372] bdevperf [2022-10-26 14:52:52.548377] --no-shconf [2022-10-26 14:52:52.548381] -l 0 [2022-10-26 14:52:52.548385] --huge-unlink [2022-10-26 14:52:52.548390] --log-level=lib.eal:6 [2022-10-26 14:52:52.548394] --log-level=lib.cryptodev:5 [2022-10-26 14:52:52.548398] --log-level=user1:6 [2022-10-26 14:52:52.548402] --iova-mode=pa [2022-10-26 14:52:52.548406] --base-virtaddr=0x200000000000 [2022-10-26 14:52:52.548410] --match-allocations [2022-10-26 14:52:52.548415] --file-prefix=spdk_pid4215 [2022-10-26 14:52:52.548422] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 14:52:52.669991] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 14:52:52.711005] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 14:52:52.711571] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_uring_foo.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_uring_foo.conf -q 4 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  284288.83 IOPS     138.81 MiB/s

 =============================================================

 Total                       :  284288.83 IOPS     138.81 MiB/s
[2022-10-26 14:53:05.541424] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 14:53:05.541465] [ DPDK EAL parameters: [2022-10-26 14:53:05.541471] bdevperf [2022-10-26 14:53:05.541475] --no-shconf [2022-10-26 14:53:05.541480] -l 0 [2022-10-26 14:53:05.541484] --huge-unlink [2022-10-26 14:53:05.541488] --log-level=lib.eal:6 [2022-10-26 14:53:05.541493] --log-level=lib.cryptodev:5 [2022-10-26 14:53:05.541497] --log-level=user1:6 [2022-10-26 14:53:05.541501] --iova-mode=pa [2022-10-26 14:53:05.541506] --base-virtaddr=0x200000000000 [2022-10-26 14:53:05.541511] --match-allocations [2022-10-26 14:53:05.541515] --file-prefix=spdk_pid4240 [2022-10-26 14:53:05.541520] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 14:53:05.663378] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 14:53:05.706987] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 14:53:05.707569] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_uring_foo.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_uring_foo.conf -q 4 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  283826.39 IOPS     138.59 MiB/s

 =============================================================

 Total                       :  283826.39 IOPS     138.59 MiB/s
[2022-10-26 14:53:18.550947] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 14:53:18.550994] [ DPDK EAL parameters: [2022-10-26 14:53:18.551000] bdevperf [2022-10-26 14:53:18.551004] --no-shconf [2022-10-26 14:53:18.551009] -l 0 [2022-10-26 14:53:18.551013] --huge-unlink [2022-10-26 14:53:18.551018] --log-level=lib.eal:6 [2022-10-26 14:53:18.551022] --log-level=lib.cryptodev:5 [2022-10-26 14:53:18.551027] --log-level=user1:6 [2022-10-26 14:53:18.551031] --iova-mode=pa [2022-10-26 14:53:18.551035] --base-virtaddr=0x200000000000 [2022-10-26 14:53:18.551039] --match-allocations [2022-10-26 14:53:18.551043] --file-prefix=spdk_pid4266 [2022-10-26 14:53:18.551048] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 14:53:18.672613] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 14:53:18.707088] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 14:53:18.707639] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_uring_foo.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_uring_foo.conf -q 4 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  283493.26 IOPS     138.42 MiB/s

 =============================================================

 Total                       :  283493.26 IOPS     138.42 MiB/s
[2022-10-26 14:53:31.638209] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 14:53:31.638256] [ DPDK EAL parameters: [2022-10-26 14:53:31.638262] bdevperf [2022-10-26 14:53:31.638266] --no-shconf [2022-10-26 14:53:31.638271] -l 0 [2022-10-26 14:53:31.638276] --huge-unlink [2022-10-26 14:53:31.638280] --log-level=lib.eal:6 [2022-10-26 14:53:31.638286] --log-level=lib.cryptodev:5 [2022-10-26 14:53:31.638290] --log-level=user1:6 [2022-10-26 14:53:31.638294] --iova-mode=pa [2022-10-26 14:53:31.638299] --base-virtaddr=0x200000000000 [2022-10-26 14:53:31.638304] --match-allocations [2022-10-26 14:53:31.638308] --file-prefix=spdk_pid4291 [2022-10-26 14:53:31.638313] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 14:53:31.760182] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 14:53:31.803079] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 14:53:31.803648] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_xnvme_io_uring.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring.conf -q 4 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  364445.67 IOPS     177.95 MiB/s

 =============================================================

 Total                       :  364445.67 IOPS     177.95 MiB/s
[2022-10-26 14:53:44.673297] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 14:53:44.673344] [ DPDK EAL parameters: [2022-10-26 14:53:44.673349] bdevperf [2022-10-26 14:53:44.673354] --no-shconf [2022-10-26 14:53:44.673358] -l 0 [2022-10-26 14:53:44.673363] --huge-unlink [2022-10-26 14:53:44.673368] --log-level=lib.eal:6 [2022-10-26 14:53:44.673373] --log-level=lib.cryptodev:5 [2022-10-26 14:53:44.673377] --log-level=user1:6 [2022-10-26 14:53:44.673381] --iova-mode=pa [2022-10-26 14:53:44.673386] --base-virtaddr=0x200000000000 [2022-10-26 14:53:44.673392] --match-allocations [2022-10-26 14:53:44.673396] --file-prefix=spdk_pid4316 [2022-10-26 14:53:44.673401] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 14:53:44.795838] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 14:53:44.839049] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 14:53:44.839650] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_xnvme_io_uring.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring.conf -q 4 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  366439.44 IOPS     178.93 MiB/s

 =============================================================

 Total                       :  366439.44 IOPS     178.93 MiB/s
[2022-10-26 14:53:57.645285] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 14:53:57.645331] [ DPDK EAL parameters: [2022-10-26 14:53:57.645337] bdevperf [2022-10-26 14:53:57.645342] --no-shconf [2022-10-26 14:53:57.645347] -l 0 [2022-10-26 14:53:57.645352] --huge-unlink [2022-10-26 14:53:57.645356] --log-level=lib.eal:6 [2022-10-26 14:53:57.645361] --log-level=lib.cryptodev:5 [2022-10-26 14:53:57.645366] --log-level=user1:6 [2022-10-26 14:53:57.645371] --iova-mode=pa [2022-10-26 14:53:57.645376] --base-virtaddr=0x200000000000 [2022-10-26 14:53:57.645381] --match-allocations [2022-10-26 14:53:57.645385] --file-prefix=spdk_pid4342 [2022-10-26 14:53:57.645390] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 14:53:57.767573] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 14:53:57.807080] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 14:53:57.807678] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_xnvme_io_uring.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring.conf -q 4 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  363187.05 IOPS     177.34 MiB/s

 =============================================================

 Total                       :  363187.05 IOPS     177.34 MiB/s
[2022-10-26 14:54:10.638786] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 14:54:10.638833] [ DPDK EAL parameters: [2022-10-26 14:54:10.638840] bdevperf [2022-10-26 14:54:10.638845] --no-shconf [2022-10-26 14:54:10.638850] -l 0 [2022-10-26 14:54:10.638854] --huge-unlink [2022-10-26 14:54:10.638858] --log-level=lib.eal:6 [2022-10-26 14:54:10.638862] --log-level=lib.cryptodev:5 [2022-10-26 14:54:10.638866] --log-level=user1:6 [2022-10-26 14:54:10.638871] --iova-mode=pa [2022-10-26 14:54:10.638876] --base-virtaddr=0x200000000000 [2022-10-26 14:54:10.638880] --match-allocations [2022-10-26 14:54:10.638884] --file-prefix=spdk_pid4368 [2022-10-26 14:54:10.638888] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 14:54:10.760857] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 14:54:10.799163] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 14:54:10.799767] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_aio_foo.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_aio_foo.conf -q 4 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  384959.50 IOPS     187.97 MiB/s

 =============================================================

 Total                       :  384959.50 IOPS     187.97 MiB/s
[2022-10-26 14:54:23.653940] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 14:54:23.653988] [ DPDK EAL parameters: [2022-10-26 14:54:23.653994] bdevperf [2022-10-26 14:54:23.653998] --no-shconf [2022-10-26 14:54:23.654003] -l 0 [2022-10-26 14:54:23.654007] --huge-unlink [2022-10-26 14:54:23.654012] --log-level=lib.eal:6 [2022-10-26 14:54:23.654017] --log-level=lib.cryptodev:5 [2022-10-26 14:54:23.654022] --log-level=user1:6 [2022-10-26 14:54:23.654026] --iova-mode=pa [2022-10-26 14:54:23.654030] --base-virtaddr=0x200000000000 [2022-10-26 14:54:23.654035] --match-allocations [2022-10-26 14:54:23.654040] --file-prefix=spdk_pid4393 [2022-10-26 14:54:23.654045] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 14:54:23.776052] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 14:54:23.828570] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 14:54:23.829162] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_aio_foo.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_aio_foo.conf -q 4 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  384611.58 IOPS     187.80 MiB/s

 =============================================================

 Total                       :  384611.58 IOPS     187.80 MiB/s
[2022-10-26 14:54:36.787399] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 14:54:36.787459] [ DPDK EAL parameters: [2022-10-26 14:54:36.787467] bdevperf [2022-10-26 14:54:36.787474] --no-shconf [2022-10-26 14:54:36.787481] -l 0 [2022-10-26 14:54:36.787486] --huge-unlink [2022-10-26 14:54:36.787491] --log-level=lib.eal:6 [2022-10-26 14:54:36.787497] --log-level=lib.cryptodev:5 [2022-10-26 14:54:36.787503] --log-level=user1:6 [2022-10-26 14:54:36.787510] --iova-mode=pa [2022-10-26 14:54:36.787516] --base-virtaddr=0x200000000000 [2022-10-26 14:54:36.787523] --match-allocations [2022-10-26 14:54:36.787530] --file-prefix=spdk_pid4418 [2022-10-26 14:54:36.787537] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 14:54:36.909401] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 14:54:36.951018] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 14:54:36.951625] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_aio_foo.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_aio_foo.conf -q 4 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  387768.06 IOPS     189.34 MiB/s

 =============================================================

 Total                       :  387768.06 IOPS     189.34 MiB/s
[2022-10-26 14:54:49.793371] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 14:54:49.793413] [ DPDK EAL parameters: [2022-10-26 14:54:49.793420] bdevperf [2022-10-26 14:54:49.793424] --no-shconf [2022-10-26 14:54:49.793429] -l 0 [2022-10-26 14:54:49.793434] --huge-unlink [2022-10-26 14:54:49.793438] --log-level=lib.eal:6 [2022-10-26 14:54:49.793444] --log-level=lib.cryptodev:5 [2022-10-26 14:54:49.793448] --log-level=user1:6 [2022-10-26 14:54:49.793452] --iova-mode=pa [2022-10-26 14:54:49.793456] --base-virtaddr=0x200000000000 [2022-10-26 14:54:49.793460] --match-allocations [2022-10-26 14:54:49.793466] --file-prefix=spdk_pid4444 [2022-10-26 14:54:49.793470] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 14:54:49.915385] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 14:54:49.955076] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 14:54:49.955675] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_xnvme_libaio.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_libaio.conf -q 4 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  336723.59 IOPS     164.42 MiB/s

 =============================================================

 Total                       :  336723.59 IOPS     164.42 MiB/s
[2022-10-26 14:55:02.826118] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 14:55:02.826165] [ DPDK EAL parameters: [2022-10-26 14:55:02.826170] bdevperf [2022-10-26 14:55:02.826176] --no-shconf [2022-10-26 14:55:02.826180] -l 0 [2022-10-26 14:55:02.826185] --huge-unlink [2022-10-26 14:55:02.826189] --log-level=lib.eal:6 [2022-10-26 14:55:02.826194] --log-level=lib.cryptodev:5 [2022-10-26 14:55:02.826198] --log-level=user1:6 [2022-10-26 14:55:02.826202] --iova-mode=pa [2022-10-26 14:55:02.826206] --base-virtaddr=0x200000000000 [2022-10-26 14:55:02.826210] --match-allocations [2022-10-26 14:55:02.826215] --file-prefix=spdk_pid4469 [2022-10-26 14:55:02.826220] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 14:55:02.948254] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 14:55:02.990987] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 14:55:02.991582] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_xnvme_libaio.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_libaio.conf -q 4 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  338676.43 IOPS     165.37 MiB/s

 =============================================================

 Total                       :  338676.43 IOPS     165.37 MiB/s
[2022-10-26 14:55:15.880904] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 14:55:15.880951] [ DPDK EAL parameters: [2022-10-26 14:55:15.880957] bdevperf [2022-10-26 14:55:15.880962] --no-shconf [2022-10-26 14:55:15.880966] -l 0 [2022-10-26 14:55:15.880971] --huge-unlink [2022-10-26 14:55:15.880976] --log-level=lib.eal:6 [2022-10-26 14:55:15.880981] --log-level=lib.cryptodev:5 [2022-10-26 14:55:15.880985] --log-level=user1:6 [2022-10-26 14:55:15.880989] --iova-mode=pa [2022-10-26 14:55:15.880995] --base-virtaddr=0x200000000000 [2022-10-26 14:55:15.881000] --match-allocations [2022-10-26 14:55:15.881004] --file-prefix=spdk_pid4494 [2022-10-26 14:55:15.881009] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 14:55:16.003039] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 14:55:16.042971] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 14:55:16.043568] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_xnvme_libaio.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_libaio.conf -q 4 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  337224.20 IOPS     164.66 MiB/s

 =============================================================

 Total                       :  337224.20 IOPS     164.66 MiB/s
[2022-10-26 14:55:29.004977] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 14:55:29.005022] [ DPDK EAL parameters: [2022-10-26 14:55:29.005028] bdevperf [2022-10-26 14:55:29.005033] --no-shconf [2022-10-26 14:55:29.005037] -l 0 [2022-10-26 14:55:29.005042] --huge-unlink [2022-10-26 14:55:29.005046] --log-level=lib.eal:6 [2022-10-26 14:55:29.005051] --log-level=lib.cryptodev:5 [2022-10-26 14:55:29.005055] --log-level=user1:6 [2022-10-26 14:55:29.005060] --iova-mode=pa [2022-10-26 14:55:29.005064] --base-virtaddr=0x200000000000 [2022-10-26 14:55:29.005068] --match-allocations [2022-10-26 14:55:29.005072] --file-prefix=spdk_pid4520 [2022-10-26 14:55:29.005077] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 14:55:29.127554] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 14:55:29.171155] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 14:55:29.171762] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_xnvme_io_uring_cmd.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring_cmd.conf -q 8 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  559323.89 IOPS     273.11 MiB/s

 =============================================================

 Total                       :  559323.89 IOPS     273.11 MiB/s
[2022-10-26 14:55:42.157720] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 14:55:42.157764] [ DPDK EAL parameters: [2022-10-26 14:55:42.157769] bdevperf [2022-10-26 14:55:42.157774] --no-shconf [2022-10-26 14:55:42.157779] -l 0 [2022-10-26 14:55:42.157784] --huge-unlink [2022-10-26 14:55:42.157788] --log-level=lib.eal:6 [2022-10-26 14:55:42.157792] --log-level=lib.cryptodev:5 [2022-10-26 14:55:42.157795] --log-level=user1:6 [2022-10-26 14:55:42.157800] --iova-mode=pa [2022-10-26 14:55:42.157804] --base-virtaddr=0x200000000000 [2022-10-26 14:55:42.157809] --match-allocations [2022-10-26 14:55:42.157813] --file-prefix=spdk_pid4546 [2022-10-26 14:55:42.157818] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 14:55:42.279857] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 14:55:42.323115] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 14:55:42.323737] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_xnvme_io_uring_cmd.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring_cmd.conf -q 8 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  629634.72 IOPS     307.44 MiB/s

 =============================================================

 Total                       :  629634.72 IOPS     307.44 MiB/s
[2022-10-26 14:55:55.451763] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 14:55:55.451810] [ DPDK EAL parameters: [2022-10-26 14:55:55.451816] bdevperf [2022-10-26 14:55:55.451821] --no-shconf [2022-10-26 14:55:55.451826] -l 0 [2022-10-26 14:55:55.451831] --huge-unlink [2022-10-26 14:55:55.451836] --log-level=lib.eal:6 [2022-10-26 14:55:55.451841] --log-level=lib.cryptodev:5 [2022-10-26 14:55:55.451845] --log-level=user1:6 [2022-10-26 14:55:55.451849] --iova-mode=pa [2022-10-26 14:55:55.451854] --base-virtaddr=0x200000000000 [2022-10-26 14:55:55.451859] --match-allocations [2022-10-26 14:55:55.451863] --file-prefix=spdk_pid4572 [2022-10-26 14:55:55.451868] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 14:55:55.573975] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 14:55:55.607138] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 14:55:55.607755] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_xnvme_io_uring_cmd.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring_cmd.conf -q 8 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  506563.20 IOPS     247.35 MiB/s

 =============================================================

 Total                       :  506563.20 IOPS     247.35 MiB/s
[2022-10-26 14:56:08.514571] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 14:56:08.514615] [ DPDK EAL parameters: [2022-10-26 14:56:08.514621] bdevperf [2022-10-26 14:56:08.514626] --no-shconf [2022-10-26 14:56:08.514631] -l 0 [2022-10-26 14:56:08.514636] --huge-unlink [2022-10-26 14:56:08.514641] --log-level=lib.eal:6 [2022-10-26 14:56:08.514645] --log-level=lib.cryptodev:5 [2022-10-26 14:56:08.514649] --log-level=user1:6 [2022-10-26 14:56:08.514653] --iova-mode=pa [2022-10-26 14:56:08.514657] --base-virtaddr=0x200000000000 [2022-10-26 14:56:08.514661] --match-allocations [2022-10-26 14:56:08.514667] --file-prefix=spdk_pid4598 [2022-10-26 14:56:08.514672] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 14:56:08.636693] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 14:56:08.679060] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 14:56:08.679646] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_uring_foo.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_uring_foo.conf -q 8 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  482815.98 IOPS     235.75 MiB/s

 =============================================================

 Total                       :  482815.98 IOPS     235.75 MiB/s
[2022-10-26 14:56:21.518478] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 14:56:21.518523] [ DPDK EAL parameters: [2022-10-26 14:56:21.518529] bdevperf [2022-10-26 14:56:21.518534] --no-shconf [2022-10-26 14:56:21.518538] -l 0 [2022-10-26 14:56:21.518543] --huge-unlink [2022-10-26 14:56:21.518547] --log-level=lib.eal:6 [2022-10-26 14:56:21.518551] --log-level=lib.cryptodev:5 [2022-10-26 14:56:21.518556] --log-level=user1:6 [2022-10-26 14:56:21.518560] --iova-mode=pa [2022-10-26 14:56:21.518564] --base-virtaddr=0x200000000000 [2022-10-26 14:56:21.518568] --match-allocations [2022-10-26 14:56:21.518575] --file-prefix=spdk_pid4623 [2022-10-26 14:56:21.518580] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 14:56:21.640578] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 14:56:21.679004] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 14:56:21.679588] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_uring_foo.conf || true

Steps uses worklets. A description of the worklet used by the current step is provided below. The description is extracted from the docstring of the worklet and provided here in verbatim / as-is. 


    Extract and transform fio and bdevperf output
    =============================================

    fio has:

    * bandwidth in KiB / second
    * latency is mean and in nano-seconds
    * IOPS are 'raw' / base-unit

    bdevperf has:

    * bandwidth in MiB / second
    * IOPS are 'raw' / base-unit

    NOTE: The metric-context need addition of backend-options, the options are
    semi-encoded by ioengine and 'label', however, that is not very precise.

{'description': '\n'
                '    Extract and transform fio and bdevperf output\n'
                '    =============================================\n'
                '\n'
                '    fio has:\n'
                '\n'
                '    * bandwidth in KiB / second\n'
                '    * latency is mean and in nano-seconds\n'
                "    * IOPS are 'raw' / base-unit\n"
                '\n'
                '    bdevperf has:\n'
                '\n'
                '    * bandwidth in MiB / second\n'
                "    * IOPS are 'raw' / base-unit\n"
                '\n'
                '    NOTE: The metric-context need addition of '
                'backend-options, the options are\n'
                "    semi-encoded by ioengine and 'label', however, that is "
                'not very precise.\n',
 'extras': {},
 'id': '002_plot_bdevperf',
 'name': 'plot_bdevperf',
 'nr': 2,
 'status': {'elapsed': 0.706589937210083,
            'failed': 0,
            'passed': 1,
            'skipped': 0,
            'started': 0.0},
 'uses': 'plot',
 'with': {'path': 'cijoe-output',
          'plot_attributes': 'plot-attributes-512.yaml',
          'tool': 'bdevperf'}}

Steps uses worklets. A description of the worklet used by the current step is provided below. The description is extracted from the docstring of the worklet and provided here in verbatim / as-is. 


    Run fio...

{'description': '\n    Run fio...\n',
 'extras': {'runlog': {'cmd_01': {'output': 'rm: cannot remove '
                                            "'/tmp/fio-output_512_1_io_uring_cmd-reference_0.txt': "
                                            'No such file or directory\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_01.output'),
                                  'state': {'begin': 1666782046.3969247,
                                            'cmd': 'rm '
                                                   '/tmp/fio-output_512_1_io_uring_cmd-reference_0.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.8351426124572754,
                                            'end': 1666782047.2320673,
                                            'err': 1,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_01.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_01.state')},
                       'cmd_02': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_02.output'),
                                  'state': {'begin': 1666782047.2342446,
                                            'cmd': '/opt/fio/bin/fio '
                                                   '--name="io_uring_cmd-reference" '
                                                   '--rw="randread" '
                                                   '--size="12G" --iodepth="1" '
                                                   '--bs="512" --direct="0" '
                                                   '--thread="1" '
                                                   '--time_based="1" '
                                                   '--runtime="7" '
                                                   '--ramp_time="3" '
                                                   '--norandommap="1" '
                                                   '--allow_file_create="0" '
                                                   '--ioengine="io_uring_cmd" '
                                                   '--filename="/dev/ng0n1" '
                                                   '--output-format="normal,json" '
                                                   '--output="/tmp/fio-output_512_1_io_uring_cmd-reference_0.txt" '
                                                   '--cpus_allowed="1"',
                                            'cwd': 'None',
                                            'elapsed': 11.069629430770874,
                                            'end': 1666782058.303874,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_02.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_02.state')},
                       'cmd_03': {'output': 'io_uring_cmd-reference: (g=0): '
                                            'rw=randread, bs=(R) 512B-512B, '
                                            '(W) 512B-512B, (T) 512B-512B, '
                                            'ioengine=io_uring_cmd, iodepth=1\n'
                                            'fio-3.32-dirty\n'
                                            'Starting 1 thread\n'
                                            '{\n'
                                            '  "fio version" : '
                                            '"fio-3.32-dirty",\n'
                                            '  "timestamp" : 1666789148,\n'
                                            '  "timestamp_ms" : '
                                            '1666789148400,\n'
                                            '  "time" : "Wed Oct 26 14:59:08 '
                                            '2022",\n'
                                            '  "jobs" : [\n'
                                            '    {\n'
                                            '      "jobname" : '
                                            '"io_uring_cmd-reference",\n'
                                            '      "groupid" : 0,\n'
                                            '      "error" : 0,\n'
                                            '      "eta" : 0,\n'
                                            '      "elapsed" : 11,\n'
                                            '      "job options" : {\n'
                                            '        "name" : '
                                            '"io_uring_cmd-reference",\n'
                                            '        "rw" : "randread",\n'
                                            '        "size" : "12G",\n'
                                            '        "iodepth" : "1",\n'
                                            '        "bs" : "512",\n'
                                            '        "direct" : "0",\n'
                                            '        "thread" : "1",\n'
                                            '        "time_based" : "1",\n'
                                            '        "runtime" : "7",\n'
                                            '        "ramp_time" : "3",\n'
                                            '        "norandommap" : "1",\n'
                                            '        "allow_file_create" : '
                                            '"0",\n'
                                            '        "ioengine" : '
                                            '"io_uring_cmd",\n'
                                            '        "filename" : '
                                            '"/dev/ng0n1",\n'
                                            '        "cpus_allowed" : "1"\n'
                                            '      },\n'
                                            '      "read" : {\n'
                                            '        "io_bytes" : 394994176,\n'
                                            '        "io_kbytes" : 385736,\n'
                                            '        "bw_bytes" : 56419679,\n'
                                            '        "bw" : 55097,\n'
                                            '        "iops" : 110194.686473,\n'
                                            '        "runtime" : 7001,\n'
                                            '        "total_ios" : 771473,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 1362,\n'
                                            '          "max" : 5961,\n'
                                            '          "mean" : 1460.212779,\n'
                                            '          "stddev" : 51.214525,\n'
                                            '          "N" : 771473\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 2955,\n'
                                            '          "max" : 117891,\n'
                                            '          "mean" : 7271.748888,\n'
                                            '          "stddev" : 971.362791,\n'
                                            '          "N" : 771473,\n'
                                            '          "percentile" : {\n'
                                            '            "1.000000" : 6752,\n'
                                            '            "5.000000" : 7008,\n'
                                            '            "10.000000" : 7008,\n'
                                            '            "20.000000" : 7072,\n'
                                            '            "30.000000" : 7072,\n'
                                            '            "40.000000" : 7136,\n'
                                            '            "50.000000" : 7200,\n'
                                            '            "60.000000" : 7200,\n'
                                            '            "70.000000" : 7264,\n'
                                            '            "80.000000" : 7264,\n'
                                            '            "90.000000" : 7392,\n'
                                            '            "95.000000" : 7456,\n'
                                            '            "99.000000" : 10432,\n'
                                            '            "99.500000" : 13632,\n'
                                            '            "99.900000" : 15680,\n'
                                            '            "99.950000" : 15808,\n'
                                            '            "99.990000" : 57600\n'
                                            '          }\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 7354,\n'
                                            '          "max" : 119324,\n'
                                            '          "mean" : 8731.961667,\n'
                                            '          "stddev" : 973.813190,\n'
                                            '          "N" : 771473\n'
                                            '        },\n'
                                            '        "bw_min" : 54956,\n'
                                            '        "bw_max" : 55331,\n'
                                            '        "bw_agg" : 100.000000,\n'
                                            '        "bw_mean" : '
                                            '55107.071429,\n'
                                            '        "bw_dev" : 106.053299,\n'
                                            '        "bw_samples" : 14,\n'
                                            '        "iops_min" : 109912,\n'
                                            '        "iops_max" : 110662,\n'
                                            '        "iops_mean" : '
                                            '110214.142857,\n'
                                            '        "iops_stddev" : '
                                            '212.106598,\n'
                                            '        "iops_samples" : 14\n'
                                            '      },\n'
                                            '      "write" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "trim" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "sync" : {\n'
                                            '        "total_ios" : 0,\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        }\n'
                                            '      },\n'
                                            '      "job_runtime" : 7000,\n'
                                            '      "usr_cpu" : 8.028571,\n'
                                            '      "sys_cpu" : 35.128571,\n'
                                            '      "ctx" : 771473,\n'
                                            '      "majf" : 0,\n'
                                            '      "minf" : 0,\n'
                                            '      "iodepth_level" : {\n'
                                            '        "1" : 100.000000,\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_submit" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_complete" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ns" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_us" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.010000,\n'
                                            '        "10" : 98.671243,\n'
                                            '        "20" : 1.310610,\n'
                                            '        "50" : 0.010000,\n'
                                            '        "100" : 0.012703,\n'
                                            '        "250" : 0.010000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ms" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000,\n'
                                            '        "2000" : 0.000000,\n'
                                            '        ">=2000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_depth" : 1,\n'
                                            '      "latency_target" : 0,\n'
                                            '      "latency_percentile" : '
                                            '100.000000,\n'
                                            '      "latency_window" : 0\n'
                                            '    }\n'
                                            '  ]\n'
                                            '}\n'
                                            '\n'
                                            'io_uring_cmd-reference: '
                                            '(groupid=0, jobs=1): err= 0: '
                                            'pid=4935: Wed Oct 26 14:59:08 '
                                            '2022\n'
                                            '  read: IOPS=110k, BW=53.8MiB/s '
                                            '(56.4MB/s)(377MiB/7001msec)\n'
                                            '    slat (nsec): min=1362, '
                                            'max=5961, avg=1460.21, '
                                            'stdev=51.21\n'
                                            '    clat (nsec): min=2955, '
                                            'max=117891, avg=7271.75, '
                                            'stdev=971.36\n'
                                            '     lat (nsec): min=7354, '
                                            'max=119324, avg=8731.96, '
                                            'stdev=973.81\n'
                                            '    clat percentiles (nsec):\n'
                                            '     |  1.00th=[ 6752],  5.00th=[ '
                                            '7008], 10.00th=[ 7008], 20.00th=[ '
                                            '7072],\n'
                                            '     | 30.00th=[ 7072], 40.00th=[ '
                                            '7136], 50.00th=[ 7200], 60.00th=[ '
                                            '7200],\n'
                                            '     | 70.00th=[ 7264], 80.00th=[ '
                                            '7264], 90.00th=[ 7392], 95.00th=[ '
                                            '7456],\n'
                                            '     | 99.00th=[10432], '
                                            '99.50th=[13632], 99.90th=[15680], '
                                            '99.95th=[15808],\n'
                                            '     | 99.99th=[57600]\n'
                                            '   bw (  KiB/s): min=54956, '
                                            'max=55331, per=100.00%, '
                                            'avg=55107.07, stdev=106.05, '
                                            'samples=14\n'
                                            '   iops        : min=109912, '
                                            'max=110662, avg=110214.14, '
                                            'stdev=212.11, samples=14\n'
                                            '  lat (usec)   : 4=0.01%, '
                                            '10=98.67%, 20=1.31%, 50=0.01%, '
                                            '100=0.01%\n'
                                            '  lat (usec)   : 250=0.01%\n'
                                            '  cpu          : usr=8.03%, '
                                            'sys=35.13%, ctx=771473, majf=0, '
                                            'minf=0\n'
                                            '  IO depths    : 1=100.0%, '
                                            '2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, >=64=0.0%\n'
                                            '     submit    : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     complete  : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     issued rwts: '
                                            'total=771473,0,0,0 short=0,0,0,0 '
                                            'dropped=0,0,0,0\n'
                                            '     latency   : target=0, '
                                            'window=0, percentile=100.00%, '
                                            'depth=1\n'
                                            '\n'
                                            'Run status group 0 (all jobs):\n'
                                            '   READ: bw=53.8MiB/s (56.4MB/s), '
                                            '53.8MiB/s-53.8MiB/s '
                                            '(56.4MB/s-56.4MB/s), io=377MiB '
                                            '(395MB), run=7001-7001msec\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_03.output'),
                                  'state': {'begin': 1666782058.306314,
                                            'cmd': 'cat '
                                                   '/tmp/fio-output_512_1_io_uring_cmd-reference_0.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.8316667079925537,
                                            'end': 1666782059.1379807,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_03.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_03.state')},
                       'cmd_04': {'output': 'rm: cannot remove '
                                            "'/tmp/fio-output_512_1_io_uring_cmd-reference_1.txt': "
                                            'No such file or directory\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_04.output'),
                                  'state': {'begin': 1666782060.2719457,
                                            'cmd': 'rm '
                                                   '/tmp/fio-output_512_1_io_uring_cmd-reference_1.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.8387961387634277,
                                            'end': 1666782061.1107419,
                                            'err': 1,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_04.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_04.state')},
                       'cmd_05': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_05.output'),
                                  'state': {'begin': 1666782061.1129358,
                                            'cmd': '/opt/fio/bin/fio '
                                                   '--name="io_uring_cmd-reference" '
                                                   '--rw="randread" '
                                                   '--size="12G" --iodepth="1" '
                                                   '--bs="512" --direct="0" '
                                                   '--thread="1" '
                                                   '--time_based="1" '
                                                   '--runtime="7" '
                                                   '--ramp_time="3" '
                                                   '--norandommap="1" '
                                                   '--allow_file_create="0" '
                                                   '--ioengine="io_uring_cmd" '
                                                   '--filename="/dev/ng0n1" '
                                                   '--output-format="normal,json" '
                                                   '--output="/tmp/fio-output_512_1_io_uring_cmd-reference_1.txt" '
                                                   '--cpus_allowed="1"',
                                            'cwd': 'None',
                                            'elapsed': 11.059051752090454,
                                            'end': 1666782072.1719875,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_05.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_05.state')},
                       'cmd_06': {'output': 'io_uring_cmd-reference: (g=0): '
                                            'rw=randread, bs=(R) 512B-512B, '
                                            '(W) 512B-512B, (T) 512B-512B, '
                                            'ioengine=io_uring_cmd, iodepth=1\n'
                                            'fio-3.32-dirty\n'
                                            'Starting 1 thread\n'
                                            '{\n'
                                            '  "fio version" : '
                                            '"fio-3.32-dirty",\n'
                                            '  "timestamp" : 1666789162,\n'
                                            '  "timestamp_ms" : '
                                            '1666789162269,\n'
                                            '  "time" : "Wed Oct 26 14:59:22 '
                                            '2022",\n'
                                            '  "jobs" : [\n'
                                            '    {\n'
                                            '      "jobname" : '
                                            '"io_uring_cmd-reference",\n'
                                            '      "groupid" : 0,\n'
                                            '      "error" : 0,\n'
                                            '      "eta" : 0,\n'
                                            '      "elapsed" : 11,\n'
                                            '      "job options" : {\n'
                                            '        "name" : '
                                            '"io_uring_cmd-reference",\n'
                                            '        "rw" : "randread",\n'
                                            '        "size" : "12G",\n'
                                            '        "iodepth" : "1",\n'
                                            '        "bs" : "512",\n'
                                            '        "direct" : "0",\n'
                                            '        "thread" : "1",\n'
                                            '        "time_based" : "1",\n'
                                            '        "runtime" : "7",\n'
                                            '        "ramp_time" : "3",\n'
                                            '        "norandommap" : "1",\n'
                                            '        "allow_file_create" : '
                                            '"0",\n'
                                            '        "ioengine" : '
                                            '"io_uring_cmd",\n'
                                            '        "filename" : '
                                            '"/dev/ng0n1",\n'
                                            '        "cpus_allowed" : "1"\n'
                                            '      },\n'
                                            '      "read" : {\n'
                                            '        "io_bytes" : 394686464,\n'
                                            '        "io_kbytes" : 385436,\n'
                                            '        "bw_bytes" : 56383780,\n'
                                            '        "bw" : 55062,\n'
                                            '        "iops" : 110124.571429,\n'
                                            '        "runtime" : 7000,\n'
                                            '        "total_ios" : 770872,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 1402,\n'
                                            '          "max" : 13765,\n'
                                            '          "mean" : 1493.597215,\n'
                                            '          "stddev" : 57.341593,\n'
                                            '          "N" : 770872\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 171,\n'
                                            '          "max" : 77646,\n'
                                            '          "mean" : 7265.220271,\n'
                                            '          "stddev" : 980.277945,\n'
                                            '          "N" : 770872,\n'
                                            '          "percentile" : {\n'
                                            '            "1.000000" : 6752,\n'
                                            '            "5.000000" : 7008,\n'
                                            '            "10.000000" : 7008,\n'
                                            '            "20.000000" : 7072,\n'
                                            '            "30.000000" : 7072,\n'
                                            '            "40.000000" : 7136,\n'
                                            '            "50.000000" : 7200,\n'
                                            '            "60.000000" : 7200,\n'
                                            '            "70.000000" : 7264,\n'
                                            '            "80.000000" : 7264,\n'
                                            '            "90.000000" : 7392,\n'
                                            '            "95.000000" : 7456,\n'
                                            '            "99.000000" : 10560,\n'
                                            '            "99.500000" : 13504,\n'
                                            '            "99.900000" : 15680,\n'
                                            '            "99.950000" : 15808,\n'
                                            '            "99.990000" : 59136\n'
                                            '          }\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 7434,\n'
                                            '          "max" : 79108,\n'
                                            '          "mean" : 8758.817486,\n'
                                            '          "stddev" : 983.505773,\n'
                                            '          "N" : 770872\n'
                                            '        },\n'
                                            '        "bw_min" : 54999,\n'
                                            '        "bw_max" : 55216,\n'
                                            '        "bw_agg" : 100.000000,\n'
                                            '        "bw_mean" : '
                                            '55079.642857,\n'
                                            '        "bw_dev" : 51.880939,\n'
                                            '        "bw_samples" : 14,\n'
                                            '        "iops_min" : 109998,\n'
                                            '        "iops_max" : 110432,\n'
                                            '        "iops_mean" : '
                                            '110159.285714,\n'
                                            '        "iops_stddev" : '
                                            '103.761879,\n'
                                            '        "iops_samples" : 14\n'
                                            '      },\n'
                                            '      "write" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "trim" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "sync" : {\n'
                                            '        "total_ios" : 0,\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        }\n'
                                            '      },\n'
                                            '      "job_runtime" : 7000,\n'
                                            '      "usr_cpu" : 6.757143,\n'
                                            '      "sys_cpu" : 36.657143,\n'
                                            '      "ctx" : 770872,\n'
                                            '      "majf" : 0,\n'
                                            '      "minf" : 0,\n'
                                            '      "iodepth_level" : {\n'
                                            '        "1" : 100.000000,\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_submit" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_complete" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ns" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.010000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_us" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.010000,\n'
                                            '        "10" : 98.658921,\n'
                                            '        "20" : 1.323436,\n'
                                            '        "50" : 0.010000,\n'
                                            '        "100" : 0.013751,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ms" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000,\n'
                                            '        "2000" : 0.000000,\n'
                                            '        ">=2000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_depth" : 1,\n'
                                            '      "latency_target" : 0,\n'
                                            '      "latency_percentile" : '
                                            '100.000000,\n'
                                            '      "latency_window" : 0\n'
                                            '    }\n'
                                            '  ]\n'
                                            '}\n'
                                            '\n'
                                            'io_uring_cmd-reference: '
                                            '(groupid=0, jobs=1): err= 0: '
                                            'pid=4981: Wed Oct 26 14:59:22 '
                                            '2022\n'
                                            '  read: IOPS=110k, BW=53.8MiB/s '
                                            '(56.4MB/s)(376MiB/7000msec)\n'
                                            '    slat (nsec): min=1402, '
                                            'max=13765, avg=1493.60, '
                                            'stdev=57.34\n'
                                            '    clat (nsec): min=171, '
                                            'max=77646, avg=7265.22, '
                                            'stdev=980.28\n'
                                            '     lat (nsec): min=7434, '
                                            'max=79108, avg=8758.82, '
                                            'stdev=983.51\n'
                                            '    clat percentiles (nsec):\n'
                                            '     |  1.00th=[ 6752],  5.00th=[ '
                                            '7008], 10.00th=[ 7008], 20.00th=[ '
                                            '7072],\n'
                                            '     | 30.00th=[ 7072], 40.00th=[ '
                                            '7136], 50.00th=[ 7200], 60.00th=[ '
                                            '7200],\n'
                                            '     | 70.00th=[ 7264], 80.00th=[ '
                                            '7264], 90.00th=[ 7392], 95.00th=[ '
                                            '7456],\n'
                                            '     | 99.00th=[10560], '
                                            '99.50th=[13504], 99.90th=[15680], '
                                            '99.95th=[15808],\n'
                                            '     | 99.99th=[59136]\n'
                                            '   bw (  KiB/s): min=54999, '
                                            'max=55216, per=100.00%, '
                                            'avg=55079.64, stdev=51.88, '
                                            'samples=14\n'
                                            '   iops        : min=109998, '
                                            'max=110432, avg=110159.29, '
                                            'stdev=103.76, samples=14\n'
                                            '  lat (nsec)   : 250=0.01%\n'
                                            '  lat (usec)   : 4=0.01%, '
                                            '10=98.66%, 20=1.32%, 50=0.01%, '
                                            '100=0.01%\n'
                                            '  cpu          : usr=6.76%, '
                                            'sys=36.66%, ctx=770872, majf=0, '
                                            'minf=0\n'
                                            '  IO depths    : 1=100.0%, '
                                            '2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, >=64=0.0%\n'
                                            '     submit    : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     complete  : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     issued rwts: '
                                            'total=770872,0,0,0 short=0,0,0,0 '
                                            'dropped=0,0,0,0\n'
                                            '     latency   : target=0, '
                                            'window=0, percentile=100.00%, '
                                            'depth=1\n'
                                            '\n'
                                            'Run status group 0 (all jobs):\n'
                                            '   READ: bw=53.8MiB/s (56.4MB/s), '
                                            '53.8MiB/s-53.8MiB/s '
                                            '(56.4MB/s-56.4MB/s), io=376MiB '
                                            '(395MB), run=7000-7000msec\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_06.output'),
                                  'state': {'begin': 1666782072.174585,
                                            'cmd': 'cat '
                                                   '/tmp/fio-output_512_1_io_uring_cmd-reference_1.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.8430523872375488,
                                            'end': 1666782073.0176375,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_06.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_06.state')},
                       'cmd_07': {'output': 'rm: cannot remove '
                                            "'/tmp/fio-output_512_1_io_uring_cmd-reference_2.txt': "
                                            'No such file or directory\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_07.output'),
                                  'state': {'begin': 1666782074.2136307,
                                            'cmd': 'rm '
                                                   '/tmp/fio-output_512_1_io_uring_cmd-reference_2.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.7940611839294434,
                                            'end': 1666782075.0076919,
                                            'err': 1,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_07.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_07.state')},
                       'cmd_08': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_08.output'),
                                  'state': {'begin': 1666782075.0098498,
                                            'cmd': '/opt/fio/bin/fio '
                                                   '--name="io_uring_cmd-reference" '
                                                   '--rw="randread" '
                                                   '--size="12G" --iodepth="1" '
                                                   '--bs="512" --direct="0" '
                                                   '--thread="1" '
                                                   '--time_based="1" '
                                                   '--runtime="7" '
                                                   '--ramp_time="3" '
                                                   '--norandommap="1" '
                                                   '--allow_file_create="0" '
                                                   '--ioengine="io_uring_cmd" '
                                                   '--filename="/dev/ng0n1" '
                                                   '--output-format="normal,json" '
                                                   '--output="/tmp/fio-output_512_1_io_uring_cmd-reference_2.txt" '
                                                   '--cpus_allowed="1"',
                                            'cwd': 'None',
                                            'elapsed': 11.038039922714233,
                                            'end': 1666782086.0478897,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_08.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_08.state')},
                       'cmd_09': {'output': 'io_uring_cmd-reference: (g=0): '
                                            'rw=randread, bs=(R) 512B-512B, '
                                            '(W) 512B-512B, (T) 512B-512B, '
                                            'ioengine=io_uring_cmd, iodepth=1\n'
                                            'fio-3.32-dirty\n'
                                            'Starting 1 thread\n'
                                            '{\n'
                                            '  "fio version" : '
                                            '"fio-3.32-dirty",\n'
                                            '  "timestamp" : 1666789176,\n'
                                            '  "timestamp_ms" : '
                                            '1666789176144,\n'
                                            '  "time" : "Wed Oct 26 14:59:36 '
                                            '2022",\n'
                                            '  "jobs" : [\n'
                                            '    {\n'
                                            '      "jobname" : '
                                            '"io_uring_cmd-reference",\n'
                                            '      "groupid" : 0,\n'
                                            '      "error" : 0,\n'
                                            '      "eta" : 0,\n'
                                            '      "elapsed" : 11,\n'
                                            '      "job options" : {\n'
                                            '        "name" : '
                                            '"io_uring_cmd-reference",\n'
                                            '        "rw" : "randread",\n'
                                            '        "size" : "12G",\n'
                                            '        "iodepth" : "1",\n'
                                            '        "bs" : "512",\n'
                                            '        "direct" : "0",\n'
                                            '        "thread" : "1",\n'
                                            '        "time_based" : "1",\n'
                                            '        "runtime" : "7",\n'
                                            '        "ramp_time" : "3",\n'
                                            '        "norandommap" : "1",\n'
                                            '        "allow_file_create" : '
                                            '"0",\n'
                                            '        "ioengine" : '
                                            '"io_uring_cmd",\n'
                                            '        "filename" : '
                                            '"/dev/ng0n1",\n'
                                            '        "cpus_allowed" : "1"\n'
                                            '      },\n'
                                            '      "read" : {\n'
                                            '        "io_bytes" : 397914112,\n'
                                            '        "io_kbytes" : 388588,\n'
                                            '        "bw_bytes" : 56836753,\n'
                                            '        "bw" : 55504,\n'
                                            '        "iops" : 111009.141551,\n'
                                            '        "runtime" : 7001,\n'
                                            '        "total_ios" : 777175,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 1352,\n'
                                            '          "max" : 16331,\n'
                                            '          "mean" : 1435.588716,\n'
                                            '          "stddev" : 64.972788,\n'
                                            '          "N" : 777176\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 200,\n'
                                            '          "max" : 118251,\n'
                                            '          "mean" : 7248.129245,\n'
                                            '          "stddev" : '
                                            '1059.290764,\n'
                                            '          "N" : 777175,\n'
                                            '          "percentile" : {\n'
                                            '            "1.000000" : 6752,\n'
                                            '            "5.000000" : 6944,\n'
                                            '            "10.000000" : 7008,\n'
                                            '            "20.000000" : 7008,\n'
                                            '            "30.000000" : 7072,\n'
                                            '            "40.000000" : 7072,\n'
                                            '            "50.000000" : 7136,\n'
                                            '            "60.000000" : 7200,\n'
                                            '            "70.000000" : 7200,\n'
                                            '            "80.000000" : 7264,\n'
                                            '            "90.000000" : 7328,\n'
                                            '            "95.000000" : 7456,\n'
                                            '            "99.000000" : 10688,\n'
                                            '            "99.500000" : 13632,\n'
                                            '            "99.900000" : 15680,\n'
                                            '            "99.950000" : 15936,\n'
                                            '            "99.990000" : 60672\n'
                                            '          }\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 7274,\n'
                                            '          "max" : 119724,\n'
                                            '          "mean" : 8683.718016,\n'
                                            '          "stddev" : '
                                            '1062.030108,\n'
                                            '          "N" : 777175\n'
                                            '        },\n'
                                            '        "bw_min" : 55395,\n'
                                            '        "bw_max" : 55748,\n'
                                            '        "bw_agg" : 100.000000,\n'
                                            '        "bw_mean" : '
                                            '55535.571429,\n'
                                            '        "bw_dev" : 121.563984,\n'
                                            '        "bw_samples" : 14,\n'
                                            '        "iops_min" : 110790,\n'
                                            '        "iops_max" : 111496,\n'
                                            '        "iops_mean" : '
                                            '111071.285714,\n'
                                            '        "iops_stddev" : '
                                            '243.041443,\n'
                                            '        "iops_samples" : 14\n'
                                            '      },\n'
                                            '      "write" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "trim" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "sync" : {\n'
                                            '        "total_ios" : 0,\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        }\n'
                                            '      },\n'
                                            '      "job_runtime" : 7000,\n'
                                            '      "usr_cpu" : 7.557143,\n'
                                            '      "sys_cpu" : 35.600000,\n'
                                            '      "ctx" : 777179,\n'
                                            '      "majf" : 0,\n'
                                            '      "minf" : 1,\n'
                                            '      "iodepth_level" : {\n'
                                            '        "1" : 100.000000,\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_submit" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_complete" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ns" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.010000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_us" : {\n'
                                            '        "2" : 0.010000,\n'
                                            '        "4" : 0.010000,\n'
                                            '        "10" : 98.510245,\n'
                                            '        "20" : 1.465307,\n'
                                            '        "50" : 0.010000,\n'
                                            '        "100" : 0.017757,\n'
                                            '        "250" : 0.010000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ms" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000,\n'
                                            '        "2000" : 0.000000,\n'
                                            '        ">=2000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_depth" : 1,\n'
                                            '      "latency_target" : 0,\n'
                                            '      "latency_percentile" : '
                                            '100.000000,\n'
                                            '      "latency_window" : 0\n'
                                            '    }\n'
                                            '  ]\n'
                                            '}\n'
                                            '\n'
                                            'io_uring_cmd-reference: '
                                            '(groupid=0, jobs=1): err= 0: '
                                            'pid=5027: Wed Oct 26 14:59:36 '
                                            '2022\n'
                                            '  read: IOPS=111k, BW=54.2MiB/s '
                                            '(56.8MB/s)(379MiB/7001msec)\n'
                                            '    slat (nsec): min=1352, '
                                            'max=16331, avg=1435.59, '
                                            'stdev=64.97\n'
                                            '    clat (nsec): min=200, '
                                            'max=118251, avg=7248.13, '
                                            'stdev=1059.29\n'
                                            '     lat (usec): min=7, max=119, '
                                            'avg= 8.68, stdev= 1.06\n'
                                            '    clat percentiles (nsec):\n'
                                            '     |  1.00th=[ 6752],  5.00th=[ '
                                            '6944], 10.00th=[ 7008], 20.00th=[ '
                                            '7008],\n'
                                            '     | 30.00th=[ 7072], 40.00th=[ '
                                            '7072], 50.00th=[ 7136], 60.00th=[ '
                                            '7200],\n'
                                            '     | 70.00th=[ 7200], 80.00th=[ '
                                            '7264], 90.00th=[ 7328], 95.00th=[ '
                                            '7456],\n'
                                            '     | 99.00th=[10688], '
                                            '99.50th=[13632], 99.90th=[15680], '
                                            '99.95th=[15936],\n'
                                            '     | 99.99th=[60672]\n'
                                            '   bw (  KiB/s): min=55395, '
                                            'max=55748, per=100.00%, '
                                            'avg=55535.57, stdev=121.56, '
                                            'samples=14\n'
                                            '   iops        : min=110790, '
                                            'max=111496, avg=111071.29, '
                                            'stdev=243.04, samples=14\n'
                                            '  lat (nsec)   : 250=0.01%\n'
                                            '  lat (usec)   : 2=0.01%, '
                                            '4=0.01%, 10=98.51%, 20=1.47%, '
                                            '50=0.01%\n'
                                            '  lat (usec)   : 100=0.02%, '
                                            '250=0.01%\n'
                                            '  cpu          : usr=7.56%, '
                                            'sys=35.60%, ctx=777179, majf=0, '
                                            'minf=1\n'
                                            '  IO depths    : 1=100.0%, '
                                            '2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, >=64=0.0%\n'
                                            '     submit    : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     complete  : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     issued rwts: '
                                            'total=777175,0,0,0 short=0,0,0,0 '
                                            'dropped=0,0,0,0\n'
                                            '     latency   : target=0, '
                                            'window=0, percentile=100.00%, '
                                            'depth=1\n'
                                            '\n'
                                            'Run status group 0 (all jobs):\n'
                                            '   READ: bw=54.2MiB/s (56.8MB/s), '
                                            '54.2MiB/s-54.2MiB/s '
                                            '(56.8MB/s-56.8MB/s), io=379MiB '
                                            '(398MB), run=7001-7001msec\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_09.output'),
                                  'state': {'begin': 1666782086.0503554,
                                            'cmd': 'cat '
                                                   '/tmp/fio-output_512_1_io_uring_cmd-reference_2.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.8288614749908447,
                                            'end': 1666782086.879217,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_09.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_09.state')},
                       'cmd_10': {'output': 'rm: cannot remove '
                                            "'/tmp/fio-output_512_1_io_uring_cmd-xnvme_0.txt': "
                                            'No such file or directory\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_10.output'),
                                  'state': {'begin': 1666782087.9278235,
                                            'cmd': 'rm '
                                                   '/tmp/fio-output_512_1_io_uring_cmd-xnvme_0.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.7727639675140381,
                                            'end': 1666782088.7005875,
                                            'err': 1,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_10.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_10.state')},
                       'cmd_100': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_512_2_io_uring_cmd-reference_0.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_100.output'),
                                   'state': {'begin': 1666782525.8023028,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_2_io_uring_cmd-reference_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7964434623718262,
                                             'end': 1666782526.5987463,
                                             'err': 1,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_100.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_100.state')},
                       'cmd_101': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_101.output'),
                                   'state': {'begin': 1666782526.6009583,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring_cmd-reference" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="2" --bs="512" '
                                                    '--direct="0" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="io_uring_cmd" '
                                                    '--filename="/dev/ng0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_2_io_uring_cmd-reference_0.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.07068920135498,
                                             'end': 1666782537.6716475,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_101.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_101.state')},
                       'cmd_102': {'output': 'io_uring_cmd-reference: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=io_uring_cmd, '
                                             'iodepth=2\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666789627,\n'
                                             '  "timestamp_ms" : '
                                             '1666789627760,\n'
                                             '  "time" : "Wed Oct 26 15:07:07 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring_cmd-reference",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring_cmd-reference",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "2",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "0",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"io_uring_cmd",\n'
                                             '        "filename" : '
                                             '"/dev/ng0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : 762248704,\n'
                                             '        "io_kbytes" : 744383,\n'
                                             '        "bw_bytes" : 108877118,\n'
                                             '        "bw" : 106325,\n'
                                             '        "iops" : 212650.478503,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 1488766,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1172,\n'
                                             '          "max" : 21300,\n'
                                             '          "mean" : 1340.914016,\n'
                                             '          "stddev" : '
                                             '138.459284,\n'
                                             '          "N" : 1488766\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 151,\n'
                                             '          "max" : 118081,\n'
                                             '          "mean" : 7781.440333,\n'
                                             '          "stddev" : '
                                             '2292.078431,\n'
                                             '          "N" : 1488767,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 5664,\n'
                                             '            "5.000000" : 5792,\n'
                                             '            "10.000000" : 5792,\n'
                                             '            "20.000000" : 5856,\n'
                                             '            "30.000000" : 5856,\n'
                                             '            "40.000000" : 5920,\n'
                                             '            "50.000000" : 9024,\n'
                                             '            "60.000000" : 9152,\n'
                                             '            "70.000000" : 9280,\n'
                                             '            "80.000000" : 9280,\n'
                                             '            "90.000000" : 9408,\n'
                                             '            "95.000000" : 9536,\n'
                                             '            "99.000000" : '
                                             '16768,\n'
                                             '            "99.500000" : '
                                             '18048,\n'
                                             '            "99.900000" : '
                                             '19072,\n'
                                             '            "99.950000" : '
                                             '23936,\n'
                                             '            "99.990000" : 64768\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 5661,\n'
                                             '          "max" : 119274,\n'
                                             '          "mean" : 9122.354383,\n'
                                             '          "stddev" : '
                                             '2373.673269,\n'
                                             '          "N" : 1488767\n'
                                             '        },\n'
                                             '        "bw_min" : 106162,\n'
                                             '        "bw_max" : 106787,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '106405.307692,\n'
                                             '        "bw_dev" : 184.502206,\n'
                                             '        "bw_samples" : 13,\n'
                                             '        "iops_min" : 212324,\n'
                                             '        "iops_max" : 213575,\n'
                                             '        "iops_mean" : '
                                             '212810.769231,\n'
                                             '        "iops_stddev" : '
                                             '369.150185,\n'
                                             '        "iops_samples" : 13\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 14.371429,\n'
                                             '      "sys_cpu" : 46.300000,\n'
                                             '      "ctx" : 746929,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 100.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 97.396434,\n'
                                             '        "20" : 2.514230,\n'
                                             '        "50" : 0.056221,\n'
                                             '        "100" : 0.025928,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 2,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring_cmd-reference: '
                                             '(groupid=0, jobs=1): err= 0: '
                                             'pid=6649: Wed Oct 26 15:07:07 '
                                             '2022\n'
                                             '  read: IOPS=213k, BW=104MiB/s '
                                             '(109MB/s)(727MiB/7001msec)\n'
                                             '    slat (nsec): min=1172, '
                                             'max=21300, avg=1340.91, '
                                             'stdev=138.46\n'
                                             '    clat (nsec): min=151, '
                                             'max=118081, avg=7781.44, '
                                             'stdev=2292.08\n'
                                             '     lat (usec): min=5, max=119, '
                                             'avg= 9.12, stdev= 2.37\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 5664],  '
                                             '5.00th=[ 5792], 10.00th=[ 5792], '
                                             '20.00th=[ 5856],\n'
                                             '     | 30.00th=[ 5856], '
                                             '40.00th=[ 5920], 50.00th=[ '
                                             '9024], 60.00th=[ 9152],\n'
                                             '     | 70.00th=[ 9280], '
                                             '80.00th=[ 9280], 90.00th=[ '
                                             '9408], 95.00th=[ 9536],\n'
                                             '     | 99.00th=[16768], '
                                             '99.50th=[18048], '
                                             '99.90th=[19072], '
                                             '99.95th=[23936],\n'
                                             '     | 99.99th=[64768]\n'
                                             '   bw (  KiB/s): min=106162, '
                                             'max=106787, per=100.00%, '
                                             'avg=106405.31, stdev=184.50, '
                                             'samples=13\n'
                                             '   iops        : min=212324, '
                                             'max=213575, avg=212810.77, '
                                             'stdev=369.15, samples=13\n'
                                             '  lat (nsec)   : 250=0.01%\n'
                                             '  lat (usec)   : 4=0.01%, '
                                             '10=97.40%, 20=2.51%, 50=0.06%, '
                                             '100=0.03%\n'
                                             '  lat (usec)   : 250=0.01%\n'
                                             '  cpu          : usr=14.37%, '
                                             'sys=46.30%, ctx=746929, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, '
                                             '2=100.0%, 4=0.0%, 8=0.0%, '
                                             '16=0.0%, 32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=1488766,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=2\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=104MiB/s (109MB/s), '
                                             '104MiB/s-104MiB/s '
                                             '(109MB/s-109MB/s), io=727MiB '
                                             '(762MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_102.output'),
                                   'state': {'begin': 1666782537.6741395,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_2_io_uring_cmd-reference_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8083879947662354,
                                             'end': 1666782538.4825275,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_102.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_102.state')},
                       'cmd_103': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_512_2_io_uring_cmd-reference_1.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_103.output'),
                                   'state': {'begin': 1666782539.5181973,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_2_io_uring_cmd-reference_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8535711765289307,
                                             'end': 1666782540.3717685,
                                             'err': 1,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_103.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_103.state')},
                       'cmd_104': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_104.output'),
                                   'state': {'begin': 1666782540.373811,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring_cmd-reference" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="2" --bs="512" '
                                                    '--direct="0" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="io_uring_cmd" '
                                                    '--filename="/dev/ng0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_2_io_uring_cmd-reference_1.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.071117401123047,
                                             'end': 1666782551.4449284,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_104.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_104.state')},
                       'cmd_105': {'output': 'io_uring_cmd-reference: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=io_uring_cmd, '
                                             'iodepth=2\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666789641,\n'
                                             '  "timestamp_ms" : '
                                             '1666789641533,\n'
                                             '  "time" : "Wed Oct 26 15:07:21 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring_cmd-reference",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring_cmd-reference",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "2",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "0",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"io_uring_cmd",\n'
                                             '        "filename" : '
                                             '"/dev/ng0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : 763893760,\n'
                                             '        "io_kbytes" : 745990,\n'
                                             '        "bw_bytes" : 109112092,\n'
                                             '        "bw" : 106554,\n'
                                             '        "iops" : 213109.270104,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 1491978,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1162,\n'
                                             '          "max" : 46527,\n'
                                             '          "mean" : 1340.729218,\n'
                                             '          "stddev" : '
                                             '143.970424,\n'
                                             '          "N" : 1491979\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 150,\n'
                                             '          "max" : 118172,\n'
                                             '          "mean" : 7760.151302,\n'
                                             '          "stddev" : '
                                             '2264.115071,\n'
                                             '          "N" : 1491979,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 5600,\n'
                                             '            "5.000000" : 5792,\n'
                                             '            "10.000000" : 5792,\n'
                                             '            "20.000000" : 5856,\n'
                                             '            "30.000000" : 5856,\n'
                                             '            "40.000000" : 5920,\n'
                                             '            "50.000000" : 8896,\n'
                                             '            "60.000000" : 9152,\n'
                                             '            "70.000000" : 9280,\n'
                                             '            "80.000000" : 9280,\n'
                                             '            "90.000000" : 9408,\n'
                                             '            "95.000000" : 9536,\n'
                                             '            "99.000000" : '
                                             '16768,\n'
                                             '            "99.500000" : '
                                             '18048,\n'
                                             '            "99.900000" : '
                                             '18816,\n'
                                             '            "99.950000" : '
                                             '23680,\n'
                                             '            "99.990000" : 60672\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 5590,\n'
                                             '          "max" : 119384,\n'
                                             '          "mean" : 9100.880520,\n'
                                             '          "stddev" : '
                                             '2350.258010,\n'
                                             '          "N" : 1491979\n'
                                             '        },\n'
                                             '        "bw_min" : 106122,\n'
                                             '        "bw_max" : 106923,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '106573.500000,\n'
                                             '        "bw_dev" : 240.951559,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 212244,\n'
                                             '        "iops_max" : 213846,\n'
                                             '        "iops_mean" : '
                                             '213147.000000,\n'
                                             '        "iops_stddev" : '
                                             '481.903118,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 14.057143,\n'
                                             '      "sys_cpu" : 46.657143,\n'
                                             '      "ctx" : 748714,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 100.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 97.423286,\n'
                                             '        "20" : 2.503053,\n'
                                             '        "50" : 0.046985,\n'
                                             '        "100" : 0.020979,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 2,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring_cmd-reference: '
                                             '(groupid=0, jobs=1): err= 0: '
                                             'pid=6695: Wed Oct 26 15:07:21 '
                                             '2022\n'
                                             '  read: IOPS=213k, BW=104MiB/s '
                                             '(109MB/s)(729MiB/7001msec)\n'
                                             '    slat (nsec): min=1162, '
                                             'max=46527, avg=1340.73, '
                                             'stdev=143.97\n'
                                             '    clat (nsec): min=150, '
                                             'max=118172, avg=7760.15, '
                                             'stdev=2264.12\n'
                                             '     lat (usec): min=5, max=119, '
                                             'avg= 9.10, stdev= 2.35\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 5600],  '
                                             '5.00th=[ 5792], 10.00th=[ 5792], '
                                             '20.00th=[ 5856],\n'
                                             '     | 30.00th=[ 5856], '
                                             '40.00th=[ 5920], 50.00th=[ '
                                             '8896], 60.00th=[ 9152],\n'
                                             '     | 70.00th=[ 9280], '
                                             '80.00th=[ 9280], 90.00th=[ '
                                             '9408], 95.00th=[ 9536],\n'
                                             '     | 99.00th=[16768], '
                                             '99.50th=[18048], '
                                             '99.90th=[18816], '
                                             '99.95th=[23680],\n'
                                             '     | 99.99th=[60672]\n'
                                             '   bw (  KiB/s): min=106122, '
                                             'max=106923, per=100.00%, '
                                             'avg=106573.50, stdev=240.95, '
                                             'samples=14\n'
                                             '   iops        : min=212244, '
                                             'max=213846, avg=213147.00, '
                                             'stdev=481.90, samples=14\n'
                                             '  lat (nsec)   : 250=0.01%\n'
                                             '  lat (usec)   : 4=0.01%, '
                                             '10=97.42%, 20=2.50%, 50=0.05%, '
                                             '100=0.02%\n'
                                             '  lat (usec)   : 250=0.01%\n'
                                             '  cpu          : usr=14.06%, '
                                             'sys=46.66%, ctx=748714, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, '
                                             '2=100.0%, 4=0.0%, 8=0.0%, '
                                             '16=0.0%, 32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=1491978,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=2\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=104MiB/s (109MB/s), '
                                             '104MiB/s-104MiB/s '
                                             '(109MB/s-109MB/s), io=729MiB '
                                             '(764MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_105.output'),
                                   'state': {'begin': 1666782551.4476588,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_2_io_uring_cmd-reference_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8775582313537598,
                                             'end': 1666782552.325217,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_105.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_105.state')},
                       'cmd_106': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_512_2_io_uring_cmd-reference_2.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_106.output'),
                                   'state': {'begin': 1666782553.3838997,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_2_io_uring_cmd-reference_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7833948135375977,
                                             'end': 1666782554.1672945,
                                             'err': 1,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_106.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_106.state')},
                       'cmd_107': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_107.output'),
                                   'state': {'begin': 1666782554.1695273,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring_cmd-reference" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="2" --bs="512" '
                                                    '--direct="0" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="io_uring_cmd" '
                                                    '--filename="/dev/ng0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_2_io_uring_cmd-reference_2.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.074486255645752,
                                             'end': 1666782565.2440135,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_107.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_107.state')},
                       'cmd_108': {'output': 'io_uring_cmd-reference: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=io_uring_cmd, '
                                             'iodepth=2\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666789655,\n'
                                             '  "timestamp_ms" : '
                                             '1666789655332,\n'
                                             '  "time" : "Wed Oct 26 15:07:35 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring_cmd-reference",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring_cmd-reference",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "2",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "0",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"io_uring_cmd",\n'
                                             '        "filename" : '
                                             '"/dev/ng0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : 764250112,\n'
                                             '        "io_kbytes" : 746338,\n'
                                             '        "bw_bytes" : 109162992,\n'
                                             '        "bw" : 106604,\n'
                                             '        "iops" : 213208.827310,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 1492675,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1162,\n'
                                             '          "max" : 25678,\n'
                                             '          "mean" : 1346.666306,\n'
                                             '          "stddev" : '
                                             '139.012440,\n'
                                             '          "N" : 1492675\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 90,\n'
                                             '          "max" : 244338,\n'
                                             '          "mean" : 7752.355304,\n'
                                             '          "stddev" : '
                                             '2295.470051,\n'
                                             '          "N" : 1492676,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 5600,\n'
                                             '            "5.000000" : 5728,\n'
                                             '            "10.000000" : 5792,\n'
                                             '            "20.000000" : 5792,\n'
                                             '            "30.000000" : 5856,\n'
                                             '            "40.000000" : 5920,\n'
                                             '            "50.000000" : 9024,\n'
                                             '            "60.000000" : 9152,\n'
                                             '            "70.000000" : 9280,\n'
                                             '            "80.000000" : 9280,\n'
                                             '            "90.000000" : 9408,\n'
                                             '            "95.000000" : 9536,\n'
                                             '            "99.000000" : '
                                             '16768,\n'
                                             '            "99.500000" : '
                                             '18048,\n'
                                             '            "99.900000" : '
                                             '18816,\n'
                                             '            "99.950000" : '
                                             '23680,\n'
                                             '            "99.990000" : 63232\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 5621,\n'
                                             '          "max" : 245951,\n'
                                             '          "mean" : 9099.021662,\n'
                                             '          "stddev" : '
                                             '2381.626970,\n'
                                             '          "N" : 1492676\n'
                                             '        },\n'
                                             '        "bw_min" : 106186,\n'
                                             '        "bw_max" : 107335,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '106623.785714,\n'
                                             '        "bw_dev" : 356.455883,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 212372,\n'
                                             '        "iops_max" : 214671,\n'
                                             '        "iops_mean" : '
                                             '213247.642857,\n'
                                             '        "iops_stddev" : '
                                             '713.065280,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 12.200000,\n'
                                             '      "sys_cpu" : 48.442857,\n'
                                             '      "ctx" : 748942,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 100.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.010000,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.010000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 97.413905,\n'
                                             '        "20" : 2.505435,\n'
                                             '        "50" : 0.049776,\n'
                                             '        "100" : 0.022912,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 2,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring_cmd-reference: '
                                             '(groupid=0, jobs=1): err= 0: '
                                             'pid=6741: Wed Oct 26 15:07:35 '
                                             '2022\n'
                                             '  read: IOPS=213k, BW=104MiB/s '
                                             '(109MB/s)(729MiB/7001msec)\n'
                                             '    slat (nsec): min=1162, '
                                             'max=25678, avg=1346.67, '
                                             'stdev=139.01\n'
                                             '    clat (nsec): min=90, '
                                             'max=244338, avg=7752.36, '
                                             'stdev=2295.47\n'
                                             '     lat (usec): min=5, max=245, '
                                             'avg= 9.10, stdev= 2.38\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 5600],  '
                                             '5.00th=[ 5728], 10.00th=[ 5792], '
                                             '20.00th=[ 5792],\n'
                                             '     | 30.00th=[ 5856], '
                                             '40.00th=[ 5920], 50.00th=[ '
                                             '9024], 60.00th=[ 9152],\n'
                                             '     | 70.00th=[ 9280], '
                                             '80.00th=[ 9280], 90.00th=[ '
                                             '9408], 95.00th=[ 9536],\n'
                                             '     | 99.00th=[16768], '
                                             '99.50th=[18048], '
                                             '99.90th=[18816], '
                                             '99.95th=[23680],\n'
                                             '     | 99.99th=[63232]\n'
                                             '   bw (  KiB/s): min=106186, '
                                             'max=107335, per=100.00%, '
                                             'avg=106623.79, stdev=356.46, '
                                             'samples=14\n'
                                             '   iops        : min=212372, '
                                             'max=214671, avg=213247.64, '
                                             'stdev=713.07, samples=14\n'
                                             '  lat (nsec)   : 100=0.01%, '
                                             '250=0.01%\n'
                                             '  lat (usec)   : 2=0.01%, '
                                             '4=0.01%, 10=97.41%, 20=2.51%, '
                                             '50=0.05%\n'
                                             '  lat (usec)   : 100=0.02%, '
                                             '250=0.01%\n'
                                             '  cpu          : usr=12.20%, '
                                             'sys=48.44%, ctx=748942, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, '
                                             '2=100.0%, 4=0.0%, 8=0.0%, '
                                             '16=0.0%, 32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=1492675,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=2\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=104MiB/s (109MB/s), '
                                             '104MiB/s-104MiB/s '
                                             '(109MB/s-109MB/s), io=729MiB '
                                             '(764MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_108.output'),
                                   'state': {'begin': 1666782565.2477517,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_2_io_uring_cmd-reference_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.838524341583252,
                                             'end': 1666782566.086276,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_108.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_108.state')},
                       'cmd_109': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_512_2_io_uring_cmd-xnvme_0.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_109.output'),
                                   'state': {'begin': 1666782567.152859,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_2_io_uring_cmd-xnvme_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8400983810424805,
                                             'end': 1666782567.9929574,
                                             'err': 1,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_109.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_109.state')},
                       'cmd_11': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_11.output'),
                                  'state': {'begin': 1666782088.7027469,
                                            'cmd': '/opt/fio/bin/fio '
                                                   '--name="io_uring_cmd-xnvme" '
                                                   '--rw="randread" '
                                                   '--size="12G" --iodepth="1" '
                                                   '--bs="512" --direct="0" '
                                                   '--thread="1" '
                                                   '--time_based="1" '
                                                   '--runtime="7" '
                                                   '--ramp_time="3" '
                                                   '--norandommap="1" '
                                                   '--allow_file_create="0" '
                                                   '--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
                                                   '--xnvme_be="linux" '
                                                   '--xnvme_async="io_uring_cmd" '
                                                   '--xnvme_sync="nvme" '
                                                   '--xnvme_admin="nvme" '
                                                   '--xnvme_dev_nsid="1" '
                                                   '--filename="/dev/ng0n1" '
                                                   '--output-format="normal,json" '
                                                   '--output="/tmp/fio-output_512_1_io_uring_cmd-xnvme_0.txt" '
                                                   '--cpus_allowed="1"',
                                            'cwd': 'None',
                                            'elapsed': 11.09110140800476,
                                            'end': 1666782099.7938483,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_11.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_11.state')},
                       'cmd_110': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_110.output'),
                                   'state': {'begin': 1666782567.9955487,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring_cmd-xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="2" --bs="512" '
                                                    '--direct="0" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
                                                    '--xnvme_be="linux" '
                                                    '--xnvme_async="io_uring_cmd" '
                                                    '--xnvme_sync="nvme" '
                                                    '--xnvme_admin="nvme" '
                                                    '--xnvme_dev_nsid="1" '
                                                    '--filename="/dev/ng0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_2_io_uring_cmd-xnvme_0.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.147419691085815,
                                             'end': 1666782579.1429684,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_110.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_110.state')},
                       'cmd_111': {'output': 'io_uring_cmd-xnvme: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=xnvme, iodepth=2\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666789669,\n'
                                             '  "timestamp_ms" : '
                                             '1666789669194,\n'
                                             '  "time" : "Wed Oct 26 15:07:49 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring_cmd-xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring_cmd-xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "2",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "0",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
                                             '        "xnvme_be" : "linux",\n'
                                             '        "xnvme_async" : '
                                             '"io_uring_cmd",\n'
                                             '        "xnvme_sync" : "nvme",\n'
                                             '        "xnvme_admin" : "nvme",\n'
                                             '        "xnvme_dev_nsid" : "1",\n'
                                             '        "filename" : '
                                             '"/dev/ng0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : 760012288,\n'
                                             '        "io_kbytes" : 742199,\n'
                                             '        "bw_bytes" : 108557675,\n'
                                             '        "bw" : 106013,\n'
                                             '        "iops" : 212026.424796,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 1484397,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1222,\n'
                                             '          "max" : 22522,\n'
                                             '          "mean" : 1423.243689,\n'
                                             '          "stddev" : '
                                             '173.920884,\n'
                                             '          "N" : 1484398\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 151,\n'
                                             '          "max" : 115626,\n'
                                             '          "mean" : 7748.459844,\n'
                                             '          "stddev" : '
                                             '2130.869047,\n'
                                             '          "N" : 1484398,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 5536,\n'
                                             '            "5.000000" : 5664,\n'
                                             '            "10.000000" : 5728,\n'
                                             '            "20.000000" : 5792,\n'
                                             '            "30.000000" : 6176,\n'
                                             '            "40.000000" : 7328,\n'
                                             '            "50.000000" : 7520,\n'
                                             '            "60.000000" : 7712,\n'
                                             '            "70.000000" : 9280,\n'
                                             '            "80.000000" : 9408,\n'
                                             '            "90.000000" : 9536,\n'
                                             '            "95.000000" : 9664,\n'
                                             '            "99.000000" : '
                                             '16768,\n'
                                             '            "99.500000" : '
                                             '18304,\n'
                                             '            "99.900000" : '
                                             '23936,\n'
                                             '            "99.950000" : '
                                             '24448,\n'
                                             '            "99.990000" : 61184\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 5751,\n'
                                             '          "max" : 116909,\n'
                                             '          "mean" : 9171.703695,\n'
                                             '          "stddev" : '
                                             '2188.738867,\n'
                                             '          "N" : 1484398\n'
                                             '        },\n'
                                             '        "bw_min" : 105599,\n'
                                             '        "bw_max" : 106438,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '106032.142857,\n'
                                             '        "bw_dev" : 187.439367,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 211198,\n'
                                             '        "iops_max" : 212876,\n'
                                             '        "iops_mean" : '
                                             '212064.285714,\n'
                                             '        "iops_stddev" : '
                                             '374.878735,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 15.985714,\n'
                                             '      "sys_cpu" : 46.657143,\n'
                                             '      "ctx" : 745370,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 1,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 100.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 97.303619,\n'
                                             '        "20" : 2.542110,\n'
                                             '        "50" : 0.122811,\n'
                                             '        "100" : 0.021288,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 2,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring_cmd-xnvme: (groupid=0, '
                                             'jobs=1): err= 0: pid=6787: Wed '
                                             'Oct 26 15:07:49 2022\n'
                                             '  read: IOPS=212k, BW=104MiB/s '
                                             '(109MB/s)(725MiB/7001msec)\n'
                                             '    slat (nsec): min=1222, '
                                             'max=22522, avg=1423.24, '
                                             'stdev=173.92\n'
                                             '    clat (nsec): min=151, '
                                             'max=115626, avg=7748.46, '
                                             'stdev=2130.87\n'
                                             '     lat (usec): min=5, max=116, '
                                             'avg= 9.17, stdev= 2.19\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 5536],  '
                                             '5.00th=[ 5664], 10.00th=[ 5728], '
                                             '20.00th=[ 5792],\n'
                                             '     | 30.00th=[ 6176], '
                                             '40.00th=[ 7328], 50.00th=[ '
                                             '7520], 60.00th=[ 7712],\n'
                                             '     | 70.00th=[ 9280], '
                                             '80.00th=[ 9408], 90.00th=[ '
                                             '9536], 95.00th=[ 9664],\n'
                                             '     | 99.00th=[16768], '
                                             '99.50th=[18304], '
                                             '99.90th=[23936], '
                                             '99.95th=[24448],\n'
                                             '     | 99.99th=[61184]\n'
                                             '   bw (  KiB/s): min=105599, '
                                             'max=106438, per=100.00%, '
                                             'avg=106032.14, stdev=187.44, '
                                             'samples=14\n'
                                             '   iops        : min=211198, '
                                             'max=212876, avg=212064.29, '
                                             'stdev=374.88, samples=14\n'
                                             '  lat (nsec)   : 250=0.01%\n'
                                             '  lat (usec)   : 4=0.01%, '
                                             '10=97.30%, 20=2.54%, 50=0.12%, '
                                             '100=0.02%\n'
                                             '  lat (usec)   : 250=0.01%\n'
                                             '  cpu          : usr=15.99%, '
                                             'sys=46.66%, ctx=745370, majf=0, '
                                             'minf=1\n'
                                             '  IO depths    : 1=0.0%, '
                                             '2=100.0%, 4=0.0%, 8=0.0%, '
                                             '16=0.0%, 32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=1484397,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=2\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=104MiB/s (109MB/s), '
                                             '104MiB/s-104MiB/s '
                                             '(109MB/s-109MB/s), io=725MiB '
                                             '(760MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_111.output'),
                                   'state': {'begin': 1666782579.1459336,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_2_io_uring_cmd-xnvme_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8462259769439697,
                                             'end': 1666782579.9921596,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_111.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_111.state')},
                       'cmd_112': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_512_2_io_uring_cmd-xnvme_1.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_112.output'),
                                   'state': {'begin': 1666782581.058541,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_2_io_uring_cmd-xnvme_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8410971164703369,
                                             'end': 1666782581.8996382,
                                             'err': 1,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_112.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_112.state')},
                       'cmd_113': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_113.output'),
                                   'state': {'begin': 1666782581.9015534,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring_cmd-xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="2" --bs="512" '
                                                    '--direct="0" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
                                                    '--xnvme_be="linux" '
                                                    '--xnvme_async="io_uring_cmd" '
                                                    '--xnvme_sync="nvme" '
                                                    '--xnvme_admin="nvme" '
                                                    '--xnvme_dev_nsid="1" '
                                                    '--filename="/dev/ng0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_2_io_uring_cmd-xnvme_1.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.12335753440857,
                                             'end': 1666782593.024911,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_113.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_113.state')},
                       'cmd_114': {'output': 'io_uring_cmd-xnvme: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=xnvme, iodepth=2\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666789683,\n'
                                             '  "timestamp_ms" : '
                                             '1666789683112,\n'
                                             '  "time" : "Wed Oct 26 15:08:03 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring_cmd-xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring_cmd-xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "2",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "0",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
                                             '        "xnvme_be" : "linux",\n'
                                             '        "xnvme_async" : '
                                             '"io_uring_cmd",\n'
                                             '        "xnvme_sync" : "nvme",\n'
                                             '        "xnvme_admin" : "nvme",\n'
                                             '        "xnvme_dev_nsid" : "1",\n'
                                             '        "filename" : '
                                             '"/dev/ng0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : 762302976,\n'
                                             '        "io_kbytes" : 744436,\n'
                                             '        "bw_bytes" : 108884870,\n'
                                             '        "bw" : 106332,\n'
                                             '        "iops" : 212665.619197,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 1488872,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1222,\n'
                                             '          "max" : 13836,\n'
                                             '          "mean" : 1386.090252,\n'
                                             '          "stddev" : '
                                             '141.744320,\n'
                                             '          "N" : 1488872\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 1954,\n'
                                             '          "max" : 79479,\n'
                                             '          "mean" : 7760.864534,\n'
                                             '          "stddev" : '
                                             '2225.390339,\n'
                                             '          "N" : 1488873,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 5536,\n'
                                             '            "5.000000" : 5728,\n'
                                             '            "10.000000" : 5728,\n'
                                             '            "20.000000" : 5792,\n'
                                             '            "30.000000" : 5856,\n'
                                             '            "40.000000" : 6496,\n'
                                             '            "50.000000" : 7520,\n'
                                             '            "60.000000" : 9152,\n'
                                             '            "70.000000" : 9280,\n'
                                             '            "80.000000" : 9408,\n'
                                             '            "90.000000" : 9408,\n'
                                             '            "95.000000" : 9536,\n'
                                             '            "99.000000" : '
                                             '16768,\n'
                                             '            "99.500000" : '
                                             '18304,\n'
                                             '            "99.900000" : '
                                             '22400,\n'
                                             '            "99.950000" : '
                                             '24192,\n'
                                             '            "99.990000" : 61696\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 5721,\n'
                                             '          "max" : 80751,\n'
                                             '          "mean" : 9146.954825,\n'
                                             '          "stddev" : '
                                             '2289.506518,\n'
                                             '          "N" : 1488873\n'
                                             '        },\n'
                                             '        "bw_min" : 106032,\n'
                                             '        "bw_max" : 106788,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '106351.571429,\n'
                                             '        "bw_dev" : 183.468176,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 212064,\n'
                                             '        "iops_max" : 213577,\n'
                                             '        "iops_mean" : '
                                             '212703.214286,\n'
                                             '        "iops_stddev" : '
                                             '367.119385,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 15.785714,\n'
                                             '      "sys_cpu" : 46.285714,\n'
                                             '      "ctx" : 747296,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 100.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.010000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 97.373313,\n'
                                             '        "20" : 2.508879,\n'
                                             '        "50" : 0.086441,\n'
                                             '        "100" : 0.022433,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 2,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring_cmd-xnvme: (groupid=0, '
                                             'jobs=1): err= 0: pid=6834: Wed '
                                             'Oct 26 15:08:03 2022\n'
                                             '  read: IOPS=213k, BW=104MiB/s '
                                             '(109MB/s)(727MiB/7001msec)\n'
                                             '    slat (nsec): min=1222, '
                                             'max=13836, avg=1386.09, '
                                             'stdev=141.74\n'
                                             '    clat (nsec): min=1954, '
                                             'max=79479, avg=7760.86, '
                                             'stdev=2225.39\n'
                                             '     lat (nsec): min=5721, '
                                             'max=80751, avg=9146.95, '
                                             'stdev=2289.51\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 5536],  '
                                             '5.00th=[ 5728], 10.00th=[ 5728], '
                                             '20.00th=[ 5792],\n'
                                             '     | 30.00th=[ 5856], '
                                             '40.00th=[ 6496], 50.00th=[ '
                                             '7520], 60.00th=[ 9152],\n'
                                             '     | 70.00th=[ 9280], '
                                             '80.00th=[ 9408], 90.00th=[ '
                                             '9408], 95.00th=[ 9536],\n'
                                             '     | 99.00th=[16768], '
                                             '99.50th=[18304], '
                                             '99.90th=[22400], '
                                             '99.95th=[24192],\n'
                                             '     | 99.99th=[61696]\n'
                                             '   bw (  KiB/s): min=106032, '
                                             'max=106788, per=100.00%, '
                                             'avg=106351.57, stdev=183.47, '
                                             'samples=14\n'
                                             '   iops        : min=212064, '
                                             'max=213577, avg=212703.21, '
                                             'stdev=367.12, samples=14\n'
                                             '  lat (usec)   : 2=0.01%, '
                                             '4=0.01%, 10=97.37%, 20=2.51%, '
                                             '50=0.09%\n'
                                             '  lat (usec)   : 100=0.02%\n'
                                             '  cpu          : usr=15.79%, '
                                             'sys=46.29%, ctx=747296, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, '
                                             '2=100.0%, 4=0.0%, 8=0.0%, '
                                             '16=0.0%, 32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=1488872,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=2\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=104MiB/s (109MB/s), '
                                             '104MiB/s-104MiB/s '
                                             '(109MB/s-109MB/s), io=727MiB '
                                             '(762MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_114.output'),
                                   'state': {'begin': 1666782593.0277514,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_2_io_uring_cmd-xnvme_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8107247352600098,
                                             'end': 1666782593.8384762,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_114.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_114.state')},
                       'cmd_115': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_512_2_io_uring_cmd-xnvme_2.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_115.output'),
                                   'state': {'begin': 1666782594.901026,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_2_io_uring_cmd-xnvme_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7831172943115234,
                                             'end': 1666782595.6841433,
                                             'err': 1,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_115.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_115.state')},
                       'cmd_116': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_116.output'),
                                   'state': {'begin': 1666782595.6861565,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring_cmd-xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="2" --bs="512" '
                                                    '--direct="0" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
                                                    '--xnvme_be="linux" '
                                                    '--xnvme_async="io_uring_cmd" '
                                                    '--xnvme_sync="nvme" '
                                                    '--xnvme_admin="nvme" '
                                                    '--xnvme_dev_nsid="1" '
                                                    '--filename="/dev/ng0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_2_io_uring_cmd-xnvme_2.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.10537338256836,
                                             'end': 1666782606.79153,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_116.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_116.state')},
                       'cmd_117': {'output': 'io_uring_cmd-xnvme: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=xnvme, iodepth=2\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666789696,\n'
                                             '  "timestamp_ms" : '
                                             '1666789696879,\n'
                                             '  "time" : "Wed Oct 26 15:08:16 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring_cmd-xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring_cmd-xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "2",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "0",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
                                             '        "xnvme_be" : "linux",\n'
                                             '        "xnvme_async" : '
                                             '"io_uring_cmd",\n'
                                             '        "xnvme_sync" : "nvme",\n'
                                             '        "xnvme_admin" : "nvme",\n'
                                             '        "xnvme_dev_nsid" : "1",\n'
                                             '        "filename" : '
                                             '"/dev/ng0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : 762843136,\n'
                                             '        "io_kbytes" : 744964,\n'
                                             '        "bw_bytes" : 108977590,\n'
                                             '        "bw" : 106423,\n'
                                             '        "iops" : 212846.714286,\n'
                                             '        "runtime" : 7000,\n'
                                             '        "total_ios" : 1489927,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1212,\n'
                                             '          "max" : 34444,\n'
                                             '          "mean" : 1402.238872,\n'
                                             '          "stddev" : '
                                             '161.624220,\n'
                                             '          "N" : 1489927\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 180,\n'
                                             '          "max" : 110517,\n'
                                             '          "mean" : 7730.380643,\n'
                                             '          "stddev" : '
                                             '2149.553208,\n'
                                             '          "N" : 1489928,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 5536,\n'
                                             '            "5.000000" : 5664,\n'
                                             '            "10.000000" : 5728,\n'
                                             '            "20.000000" : 5792,\n'
                                             '            "30.000000" : 5920,\n'
                                             '            "40.000000" : 7264,\n'
                                             '            "50.000000" : 7520,\n'
                                             '            "60.000000" : 7712,\n'
                                             '            "70.000000" : 9280,\n'
                                             '            "80.000000" : 9408,\n'
                                             '            "90.000000" : 9408,\n'
                                             '            "95.000000" : 9536,\n'
                                             '            "99.000000" : '
                                             '16768,\n'
                                             '            "99.500000" : '
                                             '18304,\n'
                                             '            "99.900000" : '
                                             '23424,\n'
                                             '            "99.950000" : '
                                             '24192,\n'
                                             '            "99.990000" : 63744\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 5690,\n'
                                             '          "max" : 111760,\n'
                                             '          "mean" : 9132.619582,\n'
                                             '          "stddev" : '
                                             '2205.875773,\n'
                                             '          "N" : 1489928\n'
                                             '        },\n'
                                             '        "bw_min" : 106063,\n'
                                             '        "bw_max" : 106683,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '106427.071429,\n'
                                             '        "bw_dev" : 168.812807,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 212126,\n'
                                             '        "iops_max" : 213366,\n'
                                             '        "iops_mean" : '
                                             '212854.142857,\n'
                                             '        "iops_stddev" : '
                                             '337.625614,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 14.314286,\n'
                                             '      "sys_cpu" : 48.085714,\n'
                                             '      "ctx" : 747909,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 100.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 97.368864,\n'
                                             '        "20" : 2.503680,\n'
                                             '        "50" : 0.096649,\n'
                                             '        "100" : 0.021276,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 2,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring_cmd-xnvme: (groupid=0, '
                                             'jobs=1): err= 0: pid=6880: Wed '
                                             'Oct 26 15:08:16 2022\n'
                                             '  read: IOPS=213k, BW=104MiB/s '
                                             '(109MB/s)(728MiB/7000msec)\n'
                                             '    slat (nsec): min=1212, '
                                             'max=34444, avg=1402.24, '
                                             'stdev=161.62\n'
                                             '    clat (nsec): min=180, '
                                             'max=110517, avg=7730.38, '
                                             'stdev=2149.55\n'
                                             '     lat (usec): min=5, max=111, '
                                             'avg= 9.13, stdev= 2.21\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 5536],  '
                                             '5.00th=[ 5664], 10.00th=[ 5728], '
                                             '20.00th=[ 5792],\n'
                                             '     | 30.00th=[ 5920], '
                                             '40.00th=[ 7264], 50.00th=[ '
                                             '7520], 60.00th=[ 7712],\n'
                                             '     | 70.00th=[ 9280], '
                                             '80.00th=[ 9408], 90.00th=[ '
                                             '9408], 95.00th=[ 9536],\n'
                                             '     | 99.00th=[16768], '
                                             '99.50th=[18304], '
                                             '99.90th=[23424], '
                                             '99.95th=[24192],\n'
                                             '     | 99.99th=[63744]\n'
                                             '   bw (  KiB/s): min=106063, '
                                             'max=106683, per=100.00%, '
                                             'avg=106427.07, stdev=168.81, '
                                             'samples=14\n'
                                             '   iops        : min=212126, '
                                             'max=213366, avg=212854.14, '
                                             'stdev=337.63, samples=14\n'
                                             '  lat (nsec)   : 250=0.01%\n'
                                             '  lat (usec)   : 4=0.01%, '
                                             '10=97.37%, 20=2.50%, 50=0.10%, '
                                             '100=0.02%\n'
                                             '  lat (usec)   : 250=0.01%\n'
                                             '  cpu          : usr=14.31%, '
                                             'sys=48.09%, ctx=747909, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, '
                                             '2=100.0%, 4=0.0%, 8=0.0%, '
                                             '16=0.0%, 32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=1489927,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=2\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=104MiB/s (109MB/s), '
                                             '104MiB/s-104MiB/s '
                                             '(109MB/s-109MB/s), io=728MiB '
                                             '(763MB), run=7000-7000msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_117.output'),
                                   'state': {'begin': 1666782606.7955425,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_2_io_uring_cmd-xnvme_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8198788166046143,
                                             'end': 1666782607.6154213,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_117.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_117.state')},
                       'cmd_118': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_512_2_io_uring_cmd-bdev_xnvme_0.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_118.output'),
                                   'state': {'begin': 1666782609.787193,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_2_io_uring_cmd-bdev_xnvme_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8587770462036133,
                                             'end': 1666782610.64597,
                                             'err': 1,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_118.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_118.state')},
                       'cmd_119': {'output': 'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_119.output'),
                                   'state': {'begin': 1666782610.6478796,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring_cmd-bdev_xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="2" --bs="512" '
                                                    '--direct="0" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="spdk_bdev" '
                                                    '--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" '
                                                    '--filename="Nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_2_io_uring_cmd-bdev_xnvme_0.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.552554607391357,
                                             'end': 1666782622.2004342,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_119.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_119.state')},
                       'cmd_12': {'output': 'io_uring_cmd-xnvme: (g=0): '
                                            'rw=randread, bs=(R) 512B-512B, '
                                            '(W) 512B-512B, (T) 512B-512B, '
                                            'ioengine=xnvme, iodepth=1\n'
                                            'fio-3.32-dirty\n'
                                            'Starting 1 thread\n'
                                            '{\n'
                                            '  "fio version" : '
                                            '"fio-3.32-dirty",\n'
                                            '  "timestamp" : 1666789189,\n'
                                            '  "timestamp_ms" : '
                                            '1666789189890,\n'
                                            '  "time" : "Wed Oct 26 14:59:49 '
                                            '2022",\n'
                                            '  "jobs" : [\n'
                                            '    {\n'
                                            '      "jobname" : '
                                            '"io_uring_cmd-xnvme",\n'
                                            '      "groupid" : 0,\n'
                                            '      "error" : 0,\n'
                                            '      "eta" : 0,\n'
                                            '      "elapsed" : 11,\n'
                                            '      "job options" : {\n'
                                            '        "name" : '
                                            '"io_uring_cmd-xnvme",\n'
                                            '        "rw" : "randread",\n'
                                            '        "size" : "12G",\n'
                                            '        "iodepth" : "1",\n'
                                            '        "bs" : "512",\n'
                                            '        "direct" : "0",\n'
                                            '        "thread" : "1",\n'
                                            '        "time_based" : "1",\n'
                                            '        "runtime" : "7",\n'
                                            '        "ramp_time" : "3",\n'
                                            '        "norandommap" : "1",\n'
                                            '        "allow_file_create" : '
                                            '"0",\n'
                                            '        "ioengine" : '
                                            '"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
                                            '        "xnvme_be" : "linux",\n'
                                            '        "xnvme_async" : '
                                            '"io_uring_cmd",\n'
                                            '        "xnvme_sync" : "nvme",\n'
                                            '        "xnvme_admin" : "nvme",\n'
                                            '        "xnvme_dev_nsid" : "1",\n'
                                            '        "filename" : '
                                            '"/dev/ng0n1",\n'
                                            '        "cpus_allowed" : "1"\n'
                                            '      },\n'
                                            '      "read" : {\n'
                                            '        "io_bytes" : 394787328,\n'
                                            '        "io_kbytes" : 385534,\n'
                                            '        "bw_bytes" : 56390133,\n'
                                            '        "bw" : 55068,\n'
                                            '        "iops" : 110136.980431,\n'
                                            '        "runtime" : 7001,\n'
                                            '        "total_ios" : 771069,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 1402,\n'
                                            '          "max" : 10500,\n'
                                            '          "mean" : 1496.382610,\n'
                                            '          "stddev" : 66.312543,\n'
                                            '          "N" : 771069\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 211,\n'
                                            '          "max" : 101751,\n'
                                            '          "mean" : 7283.762805,\n'
                                            '          "stddev" : '
                                            '1020.270135,\n'
                                            '          "N" : 771069,\n'
                                            '          "percentile" : {\n'
                                            '            "1.000000" : 6752,\n'
                                            '            "5.000000" : 7008,\n'
                                            '            "10.000000" : 7072,\n'
                                            '            "20.000000" : 7072,\n'
                                            '            "30.000000" : 7072,\n'
                                            '            "40.000000" : 7136,\n'
                                            '            "50.000000" : 7200,\n'
                                            '            "60.000000" : 7200,\n'
                                            '            "70.000000" : 7264,\n'
                                            '            "80.000000" : 7264,\n'
                                            '            "90.000000" : 7392,\n'
                                            '            "95.000000" : 7456,\n'
                                            '            "99.000000" : 10560,\n'
                                            '            "99.500000" : 13504,\n'
                                            '            "99.900000" : 15680,\n'
                                            '            "99.950000" : 15936,\n'
                                            '            "99.990000" : 58624\n'
                                            '          }\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 7324,\n'
                                            '          "max" : 103243,\n'
                                            '          "mean" : 8780.145415,\n'
                                            '          "stddev" : '
                                            '1023.065350,\n'
                                            '          "N" : 771069\n'
                                            '        },\n'
                                            '        "bw_min" : 54847,\n'
                                            '        "bw_max" : 55341,\n'
                                            '        "bw_agg" : 100.000000,\n'
                                            '        "bw_mean" : '
                                            '55095.571429,\n'
                                            '        "bw_dev" : 155.343646,\n'
                                            '        "bw_samples" : 14,\n'
                                            '        "iops_min" : 109694,\n'
                                            '        "iops_max" : 110683,\n'
                                            '        "iops_mean" : '
                                            '110191.142857,\n'
                                            '        "iops_stddev" : '
                                            '310.701404,\n'
                                            '        "iops_samples" : 14\n'
                                            '      },\n'
                                            '      "write" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "trim" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "sync" : {\n'
                                            '        "total_ios" : 0,\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        }\n'
                                            '      },\n'
                                            '      "job_runtime" : 7000,\n'
                                            '      "usr_cpu" : 10.771429,\n'
                                            '      "sys_cpu" : 33.328571,\n'
                                            '      "ctx" : 771067,\n'
                                            '      "majf" : 0,\n'
                                            '      "minf" : 0,\n'
                                            '      "iodepth_level" : {\n'
                                            '        "1" : 100.000000,\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_submit" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_complete" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ns" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.010000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_us" : {\n'
                                            '        "2" : 0.010000,\n'
                                            '        "4" : 0.010000,\n'
                                            '        "10" : 98.642275,\n'
                                            '        "20" : 1.335289,\n'
                                            '        "50" : 0.010000,\n'
                                            '        "100" : 0.016211,\n'
                                            '        "250" : 0.010000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ms" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000,\n'
                                            '        "2000" : 0.000000,\n'
                                            '        ">=2000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_depth" : 1,\n'
                                            '      "latency_target" : 0,\n'
                                            '      "latency_percentile" : '
                                            '100.000000,\n'
                                            '      "latency_window" : 0\n'
                                            '    }\n'
                                            '  ]\n'
                                            '}\n'
                                            '\n'
                                            'io_uring_cmd-xnvme: (groupid=0, '
                                            'jobs=1): err= 0: pid=5073: Wed '
                                            'Oct 26 14:59:49 2022\n'
                                            '  read: IOPS=110k, BW=53.8MiB/s '
                                            '(56.4MB/s)(376MiB/7001msec)\n'
                                            '    slat (nsec): min=1402, '
                                            'max=10500, avg=1496.38, '
                                            'stdev=66.31\n'
                                            '    clat (nsec): min=211, '
                                            'max=101751, avg=7283.76, '
                                            'stdev=1020.27\n'
                                            '     lat (usec): min=7, max=103, '
                                            'avg= 8.78, stdev= 1.02\n'
                                            '    clat percentiles (nsec):\n'
                                            '     |  1.00th=[ 6752],  5.00th=[ '
                                            '7008], 10.00th=[ 7072], 20.00th=[ '
                                            '7072],\n'
                                            '     | 30.00th=[ 7072], 40.00th=[ '
                                            '7136], 50.00th=[ 7200], 60.00th=[ '
                                            '7200],\n'
                                            '     | 70.00th=[ 7264], 80.00th=[ '
                                            '7264], 90.00th=[ 7392], 95.00th=[ '
                                            '7456],\n'
                                            '     | 99.00th=[10560], '
                                            '99.50th=[13504], 99.90th=[15680], '
                                            '99.95th=[15936],\n'
                                            '     | 99.99th=[58624]\n'
                                            '   bw (  KiB/s): min=54847, '
                                            'max=55341, per=100.00%, '
                                            'avg=55095.57, stdev=155.34, '
                                            'samples=14\n'
                                            '   iops        : min=109694, '
                                            'max=110683, avg=110191.14, '
                                            'stdev=310.70, samples=14\n'
                                            '  lat (nsec)   : 250=0.01%\n'
                                            '  lat (usec)   : 2=0.01%, '
                                            '4=0.01%, 10=98.64%, 20=1.34%, '
                                            '50=0.01%\n'
                                            '  lat (usec)   : 100=0.02%, '
                                            '250=0.01%\n'
                                            '  cpu          : usr=10.77%, '
                                            'sys=33.33%, ctx=771067, majf=0, '
                                            'minf=0\n'
                                            '  IO depths    : 1=100.0%, '
                                            '2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, >=64=0.0%\n'
                                            '     submit    : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     complete  : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     issued rwts: '
                                            'total=771069,0,0,0 short=0,0,0,0 '
                                            'dropped=0,0,0,0\n'
                                            '     latency   : target=0, '
                                            'window=0, percentile=100.00%, '
                                            'depth=1\n'
                                            '\n'
                                            'Run status group 0 (all jobs):\n'
                                            '   READ: bw=53.8MiB/s (56.4MB/s), '
                                            '53.8MiB/s-53.8MiB/s '
                                            '(56.4MB/s-56.4MB/s), io=376MiB '
                                            '(395MB), run=7001-7001msec\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_12.output'),
                                  'state': {'begin': 1666782099.7968102,
                                            'cmd': 'cat '
                                                   '/tmp/fio-output_512_1_io_uring_cmd-xnvme_0.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.8355357646942139,
                                            'end': 1666782100.632346,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_12.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_12.state')},
                       'cmd_120': {'output': 'io_uring_cmd-bdev_xnvme: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=spdk_bdev, iodepth=2\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666789712,\n'
                                             '  "timestamp_ms" : '
                                             '1666789712219,\n'
                                             '  "time" : "Wed Oct 26 15:08:32 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring_cmd-bdev_xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring_cmd-bdev_xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "2",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "0",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"spdk_bdev",\n'
                                             '        "spdk_conf" : '
                                             '"/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",\n'
                                             '        "filename" : "Nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : 909902848,\n'
                                             '        "io_kbytes" : 888577,\n'
                                             '        "bw_bytes" : 129967554,\n'
                                             '        "bw" : 126921,\n'
                                             '        "iops" : 253842.736752,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 1777153,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 220,\n'
                                             '          "max" : 16330,\n'
                                             '          "mean" : 271.297074,\n'
                                             '          "stddev" : 24.250548,\n'
                                             '          "N" : 1777153\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 281,\n'
                                             '          "max" : 121247,\n'
                                             '          "mean" : 7381.103358,\n'
                                             '          "stddev" : '
                                             '1757.802881,\n'
                                             '          "N" : 1777154,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 5984,\n'
                                             '            "5.000000" : 6240,\n'
                                             '            "10.000000" : 6368,\n'
                                             '            "20.000000" : 6624,\n'
                                             '            "30.000000" : 7072,\n'
                                             '            "40.000000" : 7136,\n'
                                             '            "50.000000" : 7200,\n'
                                             '            "60.000000" : 7264,\n'
                                             '            "70.000000" : 7328,\n'
                                             '            "80.000000" : 7520,\n'
                                             '            "90.000000" : 8032,\n'
                                             '            "95.000000" : 8256,\n'
                                             '            "99.000000" : '
                                             '16064,\n'
                                             '            "99.500000" : '
                                             '20352,\n'
                                             '            "99.900000" : '
                                             '22144,\n'
                                             '            "99.950000" : '
                                             '22400,\n'
                                             '            "99.990000" : 63232\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 5480,\n'
                                             '          "max" : 121528,\n'
                                             '          "mean" : 7652.400431,\n'
                                             '          "stddev" : '
                                             '1758.138025,\n'
                                             '          "N" : 1777154\n'
                                             '        },\n'
                                             '        "bw_min" : 125453,\n'
                                             '        "bw_max" : 129526,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '126965.500000,\n'
                                             '        "bw_dev" : 1491.929507,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 250906,\n'
                                             '        "iops_max" : 259052,\n'
                                             '        "iops_mean" : '
                                             '253931.000000,\n'
                                             '        "iops_stddev" : '
                                             '2984.065734,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 99.985714,\n'
                                             '      "sys_cpu" : 0.000000,\n'
                                             '      "ctx" : 17,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.100000,\n'
                                             '        "2" : 99.985314,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.010000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 97.055459,\n'
                                             '        "20" : 2.406096,\n'
                                             '        "50" : 0.517007,\n'
                                             '        "100" : 0.020651,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 2,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring_cmd-bdev_xnvme: '
                                             '(groupid=0, jobs=1): err= 0: '
                                             'pid=6940: Wed Oct 26 15:08:32 '
                                             '2022\n'
                                             '  read: IOPS=254k, BW=124MiB/s '
                                             '(130MB/s)(868MiB/7001msec)\n'
                                             '    slat (nsec): min=220, '
                                             'max=16330, avg=271.30, '
                                             'stdev=24.25\n'
                                             '    clat (nsec): min=281, '
                                             'max=121247, avg=7381.10, '
                                             'stdev=1757.80\n'
                                             '     lat (usec): min=5, max=121, '
                                             'avg= 7.65, stdev= 1.76\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 5984],  '
                                             '5.00th=[ 6240], 10.00th=[ 6368], '
                                             '20.00th=[ 6624],\n'
                                             '     | 30.00th=[ 7072], '
                                             '40.00th=[ 7136], 50.00th=[ '
                                             '7200], 60.00th=[ 7264],\n'
                                             '     | 70.00th=[ 7328], '
                                             '80.00th=[ 7520], 90.00th=[ '
                                             '8032], 95.00th=[ 8256],\n'
                                             '     | 99.00th=[16064], '
                                             '99.50th=[20352], '
                                             '99.90th=[22144], '
                                             '99.95th=[22400],\n'
                                             '     | 99.99th=[63232]\n'
                                             '   bw (  KiB/s): min=125453, '
                                             'max=129526, per=100.00%, '
                                             'avg=126965.50, stdev=1491.93, '
                                             'samples=14\n'
                                             '   iops        : min=250906, '
                                             'max=259052, avg=253931.00, '
                                             'stdev=2984.07, samples=14\n'
                                             '  lat (nsec)   : 500=0.01%\n'
                                             '  lat (usec)   : 4=0.01%, '
                                             '10=97.06%, 20=2.41%, 50=0.52%, '
                                             '100=0.02%\n'
                                             '  lat (usec)   : 250=0.01%\n'
                                             '  cpu          : usr=99.99%, '
                                             'sys=0.00%, ctx=17, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.1%, '
                                             '2=100.0%, 4=0.0%, 8=0.0%, '
                                             '16=0.0%, 32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=1777153,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=2\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=124MiB/s (130MB/s), '
                                             '124MiB/s-124MiB/s '
                                             '(130MB/s-130MB/s), io=868MiB '
                                             '(910MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_120.output'),
                                   'state': {'begin': 1666782622.2029004,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_2_io_uring_cmd-bdev_xnvme_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8285534381866455,
                                             'end': 1666782623.0314538,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_120.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_120.state')},
                       'cmd_121': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_512_2_io_uring_cmd-bdev_xnvme_1.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_121.output'),
                                   'state': {'begin': 1666782625.0079858,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_2_io_uring_cmd-bdev_xnvme_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8558175563812256,
                                             'end': 1666782625.8638034,
                                             'err': 1,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_121.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_121.state')},
                       'cmd_122': {'output': 'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_122.output'),
                                   'state': {'begin': 1666782625.865948,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring_cmd-bdev_xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="2" --bs="512" '
                                                    '--direct="0" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="spdk_bdev" '
                                                    '--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" '
                                                    '--filename="Nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_2_io_uring_cmd-bdev_xnvme_1.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.472501516342163,
                                             'end': 1666782637.3384495,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_122.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_122.state')},
                       'cmd_123': {'output': 'io_uring_cmd-bdev_xnvme: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=spdk_bdev, iodepth=2\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666789727,\n'
                                             '  "timestamp_ms" : '
                                             '1666789727362,\n'
                                             '  "time" : "Wed Oct 26 15:08:47 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring_cmd-bdev_xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring_cmd-bdev_xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "2",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "0",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"spdk_bdev",\n'
                                             '        "spdk_conf" : '
                                             '"/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",\n'
                                             '        "filename" : "Nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : 699008512,\n'
                                             '        "io_kbytes" : 682625,\n'
                                             '        "bw_bytes" : 99844095,\n'
                                             '        "bw" : 97503,\n'
                                             '        "iops" : 195007.856021,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 1365250,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 711,\n'
                                             '          "max" : 12062,\n'
                                             '          "mean" : 936.282706,\n'
                                             '          "stddev" : 69.667152,\n'
                                             '          "N" : 1365250\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 5841,\n'
                                             '          "max" : 82906,\n'
                                             '          "mean" : 9006.613612,\n'
                                             '          "stddev" : '
                                             '1552.777792,\n'
                                             '          "N" : 1365251,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 7264,\n'
                                             '            "5.000000" : 8032,\n'
                                             '            "10.000000" : 8512,\n'
                                             '            "20.000000" : 8640,\n'
                                             '            "30.000000" : 8768,\n'
                                             '            "40.000000" : 8768,\n'
                                             '            "50.000000" : 8768,\n'
                                             '            "60.000000" : 8896,\n'
                                             '            "70.000000" : 9024,\n'
                                             '            "80.000000" : 9024,\n'
                                             '            "90.000000" : 9280,\n'
                                             '            "95.000000" : 9536,\n'
                                             '            "99.000000" : '
                                             '17024,\n'
                                             '            "99.500000" : '
                                             '18304,\n'
                                             '            "99.900000" : '
                                             '24704,\n'
                                             '            "99.950000" : '
                                             '25728,\n'
                                             '            "99.990000" : 62208\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 7314,\n'
                                             '          "max" : 83837,\n'
                                             '          "mean" : 9942.896300,\n'
                                             '          "stddev" : '
                                             '1554.645663,\n'
                                             '          "N" : 1365251\n'
                                             '        },\n'
                                             '        "bw_min" : 97305,\n'
                                             '        "bw_max" : 98105,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '97526.428571,\n'
                                             '        "bw_dev" : 203.481889,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 194610,\n'
                                             '        "iops_max" : 196210,\n'
                                             '        "iops_mean" : '
                                             '195052.857143,\n'
                                             '        "iops_stddev" : '
                                             '406.963778,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 99.985714,\n'
                                             '      "sys_cpu" : 0.000000,\n'
                                             '      "ctx" : 16,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.534627,\n'
                                             '        "2" : 99.465373,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 96.686321,\n'
                                             '        "20" : 2.956968,\n'
                                             '        "50" : 0.336129,\n'
                                             '        "100" : 0.020656,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 2,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring_cmd-bdev_xnvme: '
                                             '(groupid=0, jobs=1): err= 0: '
                                             'pid=6998: Wed Oct 26 15:08:47 '
                                             '2022\n'
                                             '  read: IOPS=195k, BW=95.2MiB/s '
                                             '(99.8MB/s)(667MiB/7001msec)\n'
                                             '    slat (nsec): min=711, '
                                             'max=12062, avg=936.28, '
                                             'stdev=69.67\n'
                                             '    clat (nsec): min=5841, '
                                             'max=82906, avg=9006.61, '
                                             'stdev=1552.78\n'
                                             '     lat (nsec): min=7314, '
                                             'max=83837, avg=9942.90, '
                                             'stdev=1554.65\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 7264],  '
                                             '5.00th=[ 8032], 10.00th=[ 8512], '
                                             '20.00th=[ 8640],\n'
                                             '     | 30.00th=[ 8768], '
                                             '40.00th=[ 8768], 50.00th=[ '
                                             '8768], 60.00th=[ 8896],\n'
                                             '     | 70.00th=[ 9024], '
                                             '80.00th=[ 9024], 90.00th=[ '
                                             '9280], 95.00th=[ 9536],\n'
                                             '     | 99.00th=[17024], '
                                             '99.50th=[18304], '
                                             '99.90th=[24704], '
                                             '99.95th=[25728],\n'
                                             '     | 99.99th=[62208]\n'
                                             '   bw (  KiB/s): min=97305, '
                                             'max=98105, per=100.00%, '
                                             'avg=97526.43, stdev=203.48, '
                                             'samples=14\n'
                                             '   iops        : min=194610, '
                                             'max=196210, avg=195052.86, '
                                             'stdev=406.96, samples=14\n'
                                             '  lat (usec)   : 10=96.69%, '
                                             '20=2.96%, 50=0.34%, 100=0.02%\n'
                                             '  cpu          : usr=99.99%, '
                                             'sys=0.00%, ctx=16, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.5%, '
                                             '2=99.5%, 4=0.0%, 8=0.0%, '
                                             '16=0.0%, 32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=1365250,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=2\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=95.2MiB/s '
                                             '(99.8MB/s), 95.2MiB/s-95.2MiB/s '
                                             '(99.8MB/s-99.8MB/s), io=667MiB '
                                             '(699MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_123.output'),
                                   'state': {'begin': 1666782637.3410268,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_2_io_uring_cmd-bdev_xnvme_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8542912006378174,
                                             'end': 1666782638.195318,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_123.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_123.state')},
                       'cmd_124': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_512_2_io_uring_cmd-bdev_xnvme_2.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_124.output'),
                                   'state': {'begin': 1666782640.2805362,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_2_io_uring_cmd-bdev_xnvme_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8470785617828369,
                                             'end': 1666782641.1276147,
                                             'err': 1,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_124.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_124.state')},
                       'cmd_125': {'output': 'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_125.output'),
                                   'state': {'begin': 1666782641.12973,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring_cmd-bdev_xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="2" --bs="512" '
                                                    '--direct="0" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="spdk_bdev" '
                                                    '--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" '
                                                    '--filename="Nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_2_io_uring_cmd-bdev_xnvme_2.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.52648115158081,
                                             'end': 1666782652.6562111,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_125.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_125.state')},
                       'cmd_126': {'output': 'io_uring_cmd-bdev_xnvme: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=spdk_bdev, iodepth=2\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666789742,\n'
                                             '  "timestamp_ms" : '
                                             '1666789742679,\n'
                                             '  "time" : "Wed Oct 26 15:09:02 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring_cmd-bdev_xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring_cmd-bdev_xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "2",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "0",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"spdk_bdev",\n'
                                             '        "spdk_conf" : '
                                             '"/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",\n'
                                             '        "filename" : "Nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : 808450048,\n'
                                             '        "io_kbytes" : 789502,\n'
                                             '        "bw_bytes" : 115476367,\n'
                                             '        "bw" : 112769,\n'
                                             '        "iops" : 225539.637195,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 1579003,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 672,\n'
                                             '          "max" : 21169,\n'
                                             '          "mean" : 953.871964,\n'
                                             '          "stddev" : '
                                             '122.852261,\n'
                                             '          "N" : 1579003\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 752,\n'
                                             '          "max" : 110307,\n'
                                             '          "mean" : 7658.542619,\n'
                                             '          "stddev" : '
                                             '1855.562506,\n'
                                             '          "N" : 1579004,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 6624,\n'
                                             '            "5.000000" : 6752,\n'
                                             '            "10.000000" : 6816,\n'
                                             '            "20.000000" : 6944,\n'
                                             '            "30.000000" : 7072,\n'
                                             '            "40.000000" : 7136,\n'
                                             '            "50.000000" : 7200,\n'
                                             '            "60.000000" : 7264,\n'
                                             '            "70.000000" : 7392,\n'
                                             '            "80.000000" : 7520,\n'
                                             '            "90.000000" : 9536,\n'
                                             '            "95.000000" : 9792,\n'
                                             '            "99.000000" : '
                                             '17280,\n'
                                             '            "99.500000" : '
                                             '19328,\n'
                                             '            "99.900000" : '
                                             '20352,\n'
                                             '            "99.950000" : '
                                             '22912,\n'
                                             '            "99.990000" : 62208\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 7073,\n'
                                             '          "max" : 111219,\n'
                                             '          "mean" : 8612.414576,\n'
                                             '          "stddev" : '
                                             '1850.718560,\n'
                                             '          "N" : 1579004\n'
                                             '        },\n'
                                             '        "bw_min" : 112339,\n'
                                             '        "bw_max" : 113330,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '112795.214286,\n'
                                             '        "bw_dev" : 372.083361,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 224678,\n'
                                             '        "iops_max" : 226660,\n'
                                             '        "iops_mean" : '
                                             '225590.428571,\n'
                                             '        "iops_stddev" : '
                                             '744.166722,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 99.985714,\n'
                                             '      "sys_cpu" : 0.000000,\n'
                                             '      "ctx" : 12,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.296453,\n'
                                             '        "2" : 99.703547,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.010000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 96.579234,\n'
                                             '        "20" : 3.241919,\n'
                                             '        "50" : 0.156808,\n'
                                             '        "100" : 0.021849,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 2,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring_cmd-bdev_xnvme: '
                                             '(groupid=0, jobs=1): err= 0: '
                                             'pid=7056: Wed Oct 26 15:09:02 '
                                             '2022\n'
                                             '  read: IOPS=226k, BW=110MiB/s '
                                             '(115MB/s)(771MiB/7001msec)\n'
                                             '    slat (nsec): min=672, '
                                             'max=21169, avg=953.87, '
                                             'stdev=122.85\n'
                                             '    clat (nsec): min=752, '
                                             'max=110307, avg=7658.54, '
                                             'stdev=1855.56\n'
                                             '     lat (usec): min=7, max=111, '
                                             'avg= 8.61, stdev= 1.85\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 6624],  '
                                             '5.00th=[ 6752], 10.00th=[ 6816], '
                                             '20.00th=[ 6944],\n'
                                             '     | 30.00th=[ 7072], '
                                             '40.00th=[ 7136], 50.00th=[ '
                                             '7200], 60.00th=[ 7264],\n'
                                             '     | 70.00th=[ 7392], '
                                             '80.00th=[ 7520], 90.00th=[ '
                                             '9536], 95.00th=[ 9792],\n'
                                             '     | 99.00th=[17280], '
                                             '99.50th=[19328], '
                                             '99.90th=[20352], '
                                             '99.95th=[22912],\n'
                                             '     | 99.99th=[62208]\n'
                                             '   bw (  KiB/s): min=112339, '
                                             'max=113330, per=100.00%, '
                                             'avg=112795.21, stdev=372.08, '
                                             'samples=14\n'
                                             '   iops        : min=224678, '
                                             'max=226660, avg=225590.43, '
                                             'stdev=744.17, samples=14\n'
                                             '  lat (nsec)   : 1000=0.01%\n'
                                             '  lat (usec)   : 4=0.01%, '
                                             '10=96.58%, 20=3.24%, 50=0.16%, '
                                             '100=0.02%\n'
                                             '  lat (usec)   : 250=0.01%\n'
                                             '  cpu          : usr=99.99%, '
                                             'sys=0.00%, ctx=12, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.3%, '
                                             '2=99.7%, 4=0.0%, 8=0.0%, '
                                             '16=0.0%, 32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=1579003,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=2\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=110MiB/s (115MB/s), '
                                             '110MiB/s-110MiB/s '
                                             '(115MB/s-115MB/s), io=771MiB '
                                             '(808MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_126.output'),
                                   'state': {'begin': 1666782652.6588125,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_2_io_uring_cmd-bdev_xnvme_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8447775840759277,
                                             'end': 1666782653.50359,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_126.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_126.state')},
                       'cmd_127': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_512_2_io_uring-reference_0.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_127.output'),
                                   'state': {'begin': 1666782654.545266,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_2_io_uring-reference_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8357486724853516,
                                             'end': 1666782655.3810146,
                                             'err': 1,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_127.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_127.state')},
                       'cmd_128': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_128.output'),
                                   'state': {'begin': 1666782655.3829398,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring-reference" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="2" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="io_uring" '
                                                    '--filename="/dev/nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_2_io_uring-reference_0.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.060334205627441,
                                             'end': 1666782666.443274,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_128.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_128.state')},
                       'cmd_129': {'output': 'io_uring-reference: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=io_uring, iodepth=2\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666789756,\n'
                                             '  "timestamp_ms" : '
                                             '1666789756530,\n'
                                             '  "time" : "Wed Oct 26 15:09:16 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring-reference",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring-reference",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "2",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"io_uring",\n'
                                             '        "filename" : '
                                             '"/dev/nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : 763407360,\n'
                                             '        "io_kbytes" : 745515,\n'
                                             '        "bw_bytes" : 109058194,\n'
                                             '        "bw" : 106502,\n'
                                             '        "iops" : 213004.142857,\n'
                                             '        "runtime" : 7000,\n'
                                             '        "total_ios" : 1491029,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1312,\n'
                                             '          "max" : 15359,\n'
                                             '          "mean" : 1507.511643,\n'
                                             '          "stddev" : '
                                             '157.070001,\n'
                                             '          "N" : 1491029\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 1262,\n'
                                             '          "max" : 111940,\n'
                                             '          "mean" : 7604.344876,\n'
                                             '          "stddev" : '
                                             '2256.544363,\n'
                                             '          "N" : 1491030,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 5280,\n'
                                             '            "5.000000" : 5344,\n'
                                             '            "10.000000" : 5408,\n'
                                             '            "20.000000" : 5472,\n'
                                             '            "30.000000" : 5600,\n'
                                             '            "40.000000" : 7264,\n'
                                             '            "50.000000" : 7648,\n'
                                             '            "60.000000" : 8032,\n'
                                             '            "70.000000" : 9024,\n'
                                             '            "80.000000" : 9152,\n'
                                             '            "90.000000" : 9280,\n'
                                             '            "95.000000" : 9408,\n'
                                             '            "99.000000" : '
                                             '16512,\n'
                                             '            "99.500000" : '
                                             '18560,\n'
                                             '            "99.900000" : '
                                             '23936,\n'
                                             '            "99.950000" : '
                                             '24192,\n'
                                             '            "99.990000" : 62720\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 5671,\n'
                                             '          "max" : 113352,\n'
                                             '          "mean" : 9111.856576,\n'
                                             '          "stddev" : '
                                             '2313.110543,\n'
                                             '          "N" : 1491030\n'
                                             '        },\n'
                                             '        "bw_min" : 106106,\n'
                                             '        "bw_max" : 106959,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '106515.714286,\n'
                                             '        "bw_dev" : 259.918457,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 212212,\n'
                                             '        "iops_max" : 213918,\n'
                                             '        "iops_mean" : '
                                             '213031.500000,\n'
                                             '        "iops_stddev" : '
                                             '519.943007,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 12.800000,\n'
                                             '      "sys_cpu" : 50.100000,\n'
                                             '      "ctx" : 749016,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 100.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.010000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 97.365913,\n'
                                             '        "20" : 2.445761,\n'
                                             '        "50" : 0.154725,\n'
                                             '        "100" : 0.024144,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 2,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ],\n'
                                             '  "disk_util" : [\n'
                                             '    {\n'
                                             '      "name" : "nvme0n1",\n'
                                             '      "read_ios" : 2140162,\n'
                                             '      "write_ios" : 0,\n'
                                             '      "read_merges" : 0,\n'
                                             '      "write_merges" : 0,\n'
                                             '      "read_ticks" : 10972,\n'
                                             '      "write_ticks" : 0,\n'
                                             '      "in_queue" : 10972,\n'
                                             '      "util" : 99.069123\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring-reference: (groupid=0, '
                                             'jobs=1): err= 0: pid=7103: Wed '
                                             'Oct 26 15:09:16 2022\n'
                                             '  read: IOPS=213k, BW=104MiB/s '
                                             '(109MB/s)(728MiB/7000msec)\n'
                                             '    slat (nsec): min=1312, '
                                             'max=15359, avg=1507.51, '
                                             'stdev=157.07\n'
                                             '    clat (nsec): min=1262, '
                                             'max=111940, avg=7604.34, '
                                             'stdev=2256.54\n'
                                             '     lat (usec): min=5, max=113, '
                                             'avg= 9.11, stdev= 2.31\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 5280],  '
                                             '5.00th=[ 5344], 10.00th=[ 5408], '
                                             '20.00th=[ 5472],\n'
                                             '     | 30.00th=[ 5600], '
                                             '40.00th=[ 7264], 50.00th=[ '
                                             '7648], 60.00th=[ 8032],\n'
                                             '     | 70.00th=[ 9024], '
                                             '80.00th=[ 9152], 90.00th=[ '
                                             '9280], 95.00th=[ 9408],\n'
                                             '     | 99.00th=[16512], '
                                             '99.50th=[18560], '
                                             '99.90th=[23936], '
                                             '99.95th=[24192],\n'
                                             '     | 99.99th=[62720]\n'
                                             '   bw (  KiB/s): min=106106, '
                                             'max=106959, per=100.00%, '
                                             'avg=106515.71, stdev=259.92, '
                                             'samples=14\n'
                                             '   iops        : min=212212, '
                                             'max=213918, avg=213031.50, '
                                             'stdev=519.94, samples=14\n'
                                             '  lat (usec)   : 2=0.01%, '
                                             '4=0.01%, 10=97.37%, 20=2.45%, '
                                             '50=0.15%\n'
                                             '  lat (usec)   : 100=0.02%, '
                                             '250=0.01%\n'
                                             '  cpu          : usr=12.80%, '
                                             'sys=50.10%, ctx=749016, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, '
                                             '2=100.0%, 4=0.0%, 8=0.0%, '
                                             '16=0.0%, 32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=1491029,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=2\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=104MiB/s (109MB/s), '
                                             '104MiB/s-104MiB/s '
                                             '(109MB/s-109MB/s), io=728MiB '
                                             '(763MB), run=7000-7000msec\n'
                                             '\n'
                                             'Disk stats (read/write):\n'
                                             '  nvme0n1: ios=2140162/0, '
                                             'merge=0/0, ticks=10972/0, '
                                             'in_queue=10972, util=99.07%\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_129.output'),
                                   'state': {'begin': 1666782666.445754,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_2_io_uring-reference_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8473761081695557,
                                             'end': 1666782667.2931302,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_129.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_129.state')},
                       'cmd_13': {'output': 'rm: cannot remove '
                                            "'/tmp/fio-output_512_1_io_uring_cmd-xnvme_1.txt': "
                                            'No such file or directory\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_13.output'),
                                  'state': {'begin': 1666782101.6819694,
                                            'cmd': 'rm '
                                                   '/tmp/fio-output_512_1_io_uring_cmd-xnvme_1.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.8460066318511963,
                                            'end': 1666782102.527976,
                                            'err': 1,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_13.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_13.state')},
                       'cmd_130': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_512_2_io_uring-reference_1.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_130.output'),
                                   'state': {'begin': 1666782668.350055,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_2_io_uring-reference_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8273735046386719,
                                             'end': 1666782669.1774285,
                                             'err': 1,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_130.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_130.state')},
                       'cmd_131': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_131.output'),
                                   'state': {'begin': 1666782669.179487,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring-reference" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="2" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="io_uring" '
                                                    '--filename="/dev/nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_2_io_uring-reference_1.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.059387683868408,
                                             'end': 1666782680.2388747,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_131.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_131.state')},
                       'cmd_132': {'output': 'io_uring-reference: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=io_uring, iodepth=2\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666789770,\n'
                                             '  "timestamp_ms" : '
                                             '1666789770326,\n'
                                             '  "time" : "Wed Oct 26 15:09:30 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring-reference",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring-reference",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "2",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"io_uring",\n'
                                             '        "filename" : '
                                             '"/dev/nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : 757835776,\n'
                                             '        "io_kbytes" : 740074,\n'
                                             '        "bw_bytes" : 108246789,\n'
                                             '        "bw" : 105709,\n'
                                             '        "iops" : 211419.368662,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 1480147,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1332,\n'
                                             '          "max" : 26660,\n'
                                             '          "mean" : 1516.955765,\n'
                                             '          "stddev" : '
                                             '165.330542,\n'
                                             '          "N" : 1480147\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 200,\n'
                                             '          "max" : 109896,\n'
                                             '          "mean" : 7661.495982,\n'
                                             '          "stddev" : '
                                             '2256.062018,\n'
                                             '          "N" : 1480148,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 5280,\n'
                                             '            "5.000000" : 5408,\n'
                                             '            "10.000000" : 5472,\n'
                                             '            "20.000000" : 5536,\n'
                                             '            "30.000000" : 5792,\n'
                                             '            "40.000000" : 7328,\n'
                                             '            "50.000000" : 7648,\n'
                                             '            "60.000000" : 7968,\n'
                                             '            "70.000000" : 9152,\n'
                                             '            "80.000000" : 9152,\n'
                                             '            "90.000000" : 9280,\n'
                                             '            "95.000000" : 9536,\n'
                                             '            "99.000000" : '
                                             '16768,\n'
                                             '            "99.500000" : '
                                             '18560,\n'
                                             '            "99.900000" : '
                                             '23936,\n'
                                             '            "99.950000" : '
                                             '24448,\n'
                                             '            "99.990000" : 65280\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 5771,\n'
                                             '          "max" : 111358,\n'
                                             '          "mean" : 9178.451778,\n'
                                             '          "stddev" : '
                                             '2304.102186,\n'
                                             '          "N" : 1480148\n'
                                             '        },\n'
                                             '        "bw_min" : 105505,\n'
                                             '        "bw_max" : 106062,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '105778.000000,\n'
                                             '        "bw_dev" : 128.192764,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 211010,\n'
                                             '        "iops_max" : 212124,\n'
                                             '        "iops_mean" : '
                                             '211556.142857,\n'
                                             '        "iops_stddev" : '
                                             '256.392385,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 13.585714,\n'
                                             '      "sys_cpu" : 49.757143,\n'
                                             '      "ctx" : 743789,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 100.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 97.308105,\n'
                                             '        "20" : 2.463809,\n'
                                             '        "50" : 0.192211,\n'
                                             '        "100" : 0.026146,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 2,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ],\n'
                                             '  "disk_util" : [\n'
                                             '    {\n'
                                             '      "name" : "nvme0n1",\n'
                                             '      "read_ios" : 2124569,\n'
                                             '      "write_ios" : 0,\n'
                                             '      "read_merges" : 0,\n'
                                             '      "write_merges" : 0,\n'
                                             '      "read_ticks" : 10947,\n'
                                             '      "write_ticks" : 0,\n'
                                             '      "in_queue" : 10947,\n'
                                             '      "util" : 99.068754\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring-reference: (groupid=0, '
                                             'jobs=1): err= 0: pid=7150: Wed '
                                             'Oct 26 15:09:30 2022\n'
                                             '  read: IOPS=211k, BW=103MiB/s '
                                             '(108MB/s)(723MiB/7001msec)\n'
                                             '    slat (nsec): min=1332, '
                                             'max=26660, avg=1516.96, '
                                             'stdev=165.33\n'
                                             '    clat (nsec): min=200, '
                                             'max=109896, avg=7661.50, '
                                             'stdev=2256.06\n'
                                             '     lat (usec): min=5, max=111, '
                                             'avg= 9.18, stdev= 2.30\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 5280],  '
                                             '5.00th=[ 5408], 10.00th=[ 5472], '
                                             '20.00th=[ 5536],\n'
                                             '     | 30.00th=[ 5792], '
                                             '40.00th=[ 7328], 50.00th=[ '
                                             '7648], 60.00th=[ 7968],\n'
                                             '     | 70.00th=[ 9152], '
                                             '80.00th=[ 9152], 90.00th=[ '
                                             '9280], 95.00th=[ 9536],\n'
                                             '     | 99.00th=[16768], '
                                             '99.50th=[18560], '
                                             '99.90th=[23936], '
                                             '99.95th=[24448],\n'
                                             '     | 99.99th=[65280]\n'
                                             '   bw (  KiB/s): min=105505, '
                                             'max=106062, per=100.00%, '
                                             'avg=105778.00, stdev=128.19, '
                                             'samples=14\n'
                                             '   iops        : min=211010, '
                                             'max=212124, avg=211556.14, '
                                             'stdev=256.39, samples=14\n'
                                             '  lat (nsec)   : 250=0.01%\n'
                                             '  lat (usec)   : 4=0.01%, '
                                             '10=97.31%, 20=2.46%, 50=0.19%, '
                                             '100=0.03%\n'
                                             '  lat (usec)   : 250=0.01%\n'
                                             '  cpu          : usr=13.59%, '
                                             'sys=49.76%, ctx=743789, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, '
                                             '2=100.0%, 4=0.0%, 8=0.0%, '
                                             '16=0.0%, 32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=1480147,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=2\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=103MiB/s (108MB/s), '
                                             '103MiB/s-103MiB/s '
                                             '(108MB/s-108MB/s), io=723MiB '
                                             '(758MB), run=7001-7001msec\n'
                                             '\n'
                                             'Disk stats (read/write):\n'
                                             '  nvme0n1: ios=2124569/0, '
                                             'merge=0/0, ticks=10947/0, '
                                             'in_queue=10947, util=99.07%\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_132.output'),
                                   'state': {'begin': 1666782680.2413769,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_2_io_uring-reference_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8027191162109375,
                                             'end': 1666782681.044096,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_132.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_132.state')},
                       'cmd_133': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_512_2_io_uring-reference_2.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_133.output'),
                                   'state': {'begin': 1666782682.1035376,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_2_io_uring-reference_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.838510274887085,
                                             'end': 1666782682.9420478,
                                             'err': 1,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_133.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_133.state')},
                       'cmd_134': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_134.output'),
                                   'state': {'begin': 1666782682.9488225,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring-reference" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="2" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="io_uring" '
                                                    '--filename="/dev/nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_2_io_uring-reference_2.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.067566871643066,
                                             'end': 1666782694.0163894,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_134.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_134.state')},
                       'cmd_135': {'output': 'io_uring-reference: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=io_uring, iodepth=2\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666789784,\n'
                                             '  "timestamp_ms" : '
                                             '1666789784103,\n'
                                             '  "time" : "Wed Oct 26 15:09:44 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring-reference",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring-reference",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "2",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"io_uring",\n'
                                             '        "filename" : '
                                             '"/dev/nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : 760478720,\n'
                                             '        "io_kbytes" : 742655,\n'
                                             '        "bw_bytes" : 108624299,\n'
                                             '        "bw" : 106078,\n'
                                             '        "iops" : 212156.691901,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 1485309,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1332,\n'
                                             '          "max" : 22673,\n'
                                             '          "mean" : 1514.181856,\n'
                                             '          "stddev" : '
                                             '153.990339,\n'
                                             '          "N" : 1485309\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 100,\n'
                                             '          "max" : 117761,\n'
                                             '          "mean" : 7616.194574,\n'
                                             '          "stddev" : '
                                             '2211.320112,\n'
                                             '          "N" : 1485310,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 5280,\n'
                                             '            "5.000000" : 5344,\n'
                                             '            "10.000000" : 5344,\n'
                                             '            "20.000000" : 5472,\n'
                                             '            "30.000000" : 5664,\n'
                                             '            "40.000000" : 7328,\n'
                                             '            "50.000000" : 7648,\n'
                                             '            "60.000000" : 7968,\n'
                                             '            "70.000000" : 9024,\n'
                                             '            "80.000000" : 9152,\n'
                                             '            "90.000000" : 9280,\n'
                                             '            "95.000000" : 9408,\n'
                                             '            "99.000000" : '
                                             '16512,\n'
                                             '            "99.500000" : '
                                             '18560,\n'
                                             '            "99.900000" : '
                                             '23680,\n'
                                             '            "99.950000" : '
                                             '24192,\n'
                                             '            "99.990000" : 59648\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 5700,\n'
                                             '          "max" : 119204,\n'
                                             '          "mean" : 9130.376490,\n'
                                             '          "stddev" : '
                                             '2257.397056,\n'
                                             '          "N" : 1485310\n'
                                             '        },\n'
                                             '        "bw_min" : 105355,\n'
                                             '        "bw_max" : 106441,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '106110.214286,\n'
                                             '        "bw_dev" : 316.062598,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 210711,\n'
                                             '        "iops_max" : 212882,\n'
                                             '        "iops_mean" : '
                                             '212220.571429,\n'
                                             '        "iops_stddev" : '
                                             '631.966980,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 13.228571,\n'
                                             '      "sys_cpu" : 50.228571,\n'
                                             '      "ctx" : 746364,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 100.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 97.327155,\n'
                                             '        "20" : 2.466490,\n'
                                             '        "50" : 0.177135,\n'
                                             '        "100" : 0.019592,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 2,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ],\n'
                                             '  "disk_util" : [\n'
                                             '    {\n'
                                             '      "name" : "nvme0n1",\n'
                                             '      "read_ios" : 2126963,\n'
                                             '      "write_ios" : 0,\n'
                                             '      "read_merges" : 0,\n'
                                             '      "write_merges" : 0,\n'
                                             '      "read_ticks" : 10923,\n'
                                             '      "write_ticks" : 0,\n'
                                             '      "in_queue" : 10923,\n'
                                             '      "util" : 99.078934\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring-reference: (groupid=0, '
                                             'jobs=1): err= 0: pid=7196: Wed '
                                             'Oct 26 15:09:44 2022\n'
                                             '  read: IOPS=212k, BW=104MiB/s '
                                             '(109MB/s)(725MiB/7001msec)\n'
                                             '    slat (nsec): min=1332, '
                                             'max=22673, avg=1514.18, '
                                             'stdev=153.99\n'
                                             '    clat (nsec): min=100, '
                                             'max=117761, avg=7616.19, '
                                             'stdev=2211.32\n'
                                             '     lat (usec): min=5, max=119, '
                                             'avg= 9.13, stdev= 2.26\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 5280],  '
                                             '5.00th=[ 5344], 10.00th=[ 5344], '
                                             '20.00th=[ 5472],\n'
                                             '     | 30.00th=[ 5664], '
                                             '40.00th=[ 7328], 50.00th=[ '
                                             '7648], 60.00th=[ 7968],\n'
                                             '     | 70.00th=[ 9024], '
                                             '80.00th=[ 9152], 90.00th=[ '
                                             '9280], 95.00th=[ 9408],\n'
                                             '     | 99.00th=[16512], '
                                             '99.50th=[18560], '
                                             '99.90th=[23680], '
                                             '99.95th=[24192],\n'
                                             '     | 99.99th=[59648]\n'
                                             '   bw (  KiB/s): min=105355, '
                                             'max=106441, per=100.00%, '
                                             'avg=106110.21, stdev=316.06, '
                                             'samples=14\n'
                                             '   iops        : min=210711, '
                                             'max=212882, avg=212220.57, '
                                             'stdev=631.97, samples=14\n'
                                             '  lat (nsec)   : 250=0.01%\n'
                                             '  lat (usec)   : 4=0.01%, '
                                             '10=97.33%, 20=2.47%, 50=0.18%, '
                                             '100=0.02%\n'
                                             '  lat (usec)   : 250=0.01%\n'
                                             '  cpu          : usr=13.23%, '
                                             'sys=50.23%, ctx=746364, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, '
                                             '2=100.0%, 4=0.0%, 8=0.0%, '
                                             '16=0.0%, 32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=1485309,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=2\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=104MiB/s (109MB/s), '
                                             '104MiB/s-104MiB/s '
                                             '(109MB/s-109MB/s), io=725MiB '
                                             '(760MB), run=7001-7001msec\n'
                                             '\n'
                                             'Disk stats (read/write):\n'
                                             '  nvme0n1: ios=2126963/0, '
                                             'merge=0/0, ticks=10923/0, '
                                             'in_queue=10923, util=99.08%\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_135.output'),
                                   'state': {'begin': 1666782694.018831,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_2_io_uring-reference_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8572797775268555,
                                             'end': 1666782694.8761108,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_135.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_135.state')},
                       'cmd_136': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_512_2_io_uring-bdev_uring_0.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_136.output'),
                                   'state': {'begin': 1666782696.9613917,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_2_io_uring-bdev_uring_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7837743759155273,
                                             'end': 1666782697.745166,
                                             'err': 1,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_136.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_136.state')},
                       'cmd_137': {'output': 'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_137.output'),
                                   'state': {'begin': 1666782697.7469563,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring-bdev_uring" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="2" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="spdk_bdev" '
                                                    '--spdk_conf="/tmp/spdk_bdev_uring_foo.conf" '
                                                    '--filename="Nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_2_io_uring-bdev_uring_0.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.493335962295532,
                                             'end': 1666782709.2402923,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_137.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_137.state')},
                       'cmd_138': {'output': 'io_uring-bdev_uring: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=spdk_bdev, iodepth=2\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666789799,\n'
                                             '  "timestamp_ms" : '
                                             '1666789799256,\n'
                                             '  "time" : "Wed Oct 26 15:09:59 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring-bdev_uring",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring-bdev_uring",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "2",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"spdk_bdev",\n'
                                             '        "spdk_conf" : '
                                             '"/tmp/spdk_bdev_uring_foo.conf",\n'
                                             '        "filename" : "Nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : 928209408,\n'
                                             '        "io_kbytes" : 906454,\n'
                                             '        "bw_bytes" : 132582403,\n'
                                             '        "bw" : 129475,\n'
                                             '        "iops" : 258949.864305,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 1812908,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 110,\n'
                                             '          "max" : 12924,\n'
                                             '          "mean" : 173.489903,\n'
                                             '          "stddev" : 32.365104,\n'
                                             '          "N" : 1812908\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 5941,\n'
                                             '          "max" : 118202,\n'
                                             '          "mean" : 7276.558786,\n'
                                             '          "stddev" : '
                                             '1678.838912,\n'
                                             '          "N" : 1812909,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 6560,\n'
                                             '            "5.000000" : 6688,\n'
                                             '            "10.000000" : 6688,\n'
                                             '            "20.000000" : 6752,\n'
                                             '            "30.000000" : 6752,\n'
                                             '            "40.000000" : 6816,\n'
                                             '            "50.000000" : 6880,\n'
                                             '            "60.000000" : 6880,\n'
                                             '            "70.000000" : 6944,\n'
                                             '            "80.000000" : 7200,\n'
                                             '            "90.000000" : 8256,\n'
                                             '            "95.000000" : 8384,\n'
                                             '            "99.000000" : '
                                             '14656,\n'
                                             '            "99.500000" : '
                                             '18560,\n'
                                             '            "99.900000" : '
                                             '20352,\n'
                                             '            "99.950000" : '
                                             '21120,\n'
                                             '            "99.990000" : 62720\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 6111,\n'
                                             '          "max" : 118372,\n'
                                             '          "mean" : 7450.048687,\n'
                                             '          "stddev" : '
                                             '1675.358061,\n'
                                             '          "N" : 1812909\n'
                                             '        },\n'
                                             '        "bw_min" : 128981,\n'
                                             '        "bw_max" : 130259,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '129499.928571,\n'
                                             '        "bw_dev" : 474.433989,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 257963,\n'
                                             '        "iops_max" : 260518,\n'
                                             '        "iops_mean" : '
                                             '258999.928571,\n'
                                             '        "iops_stddev" : '
                                             '948.783875,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 32.957143,\n'
                                             '      "sys_cpu" : 67.014286,\n'
                                             '      "ctx" : 11,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 8.368213,\n'
                                             '        "2" : 91.631787,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 97.055173,\n'
                                             '        "20" : 2.794626,\n'
                                             '        "50" : 0.126316,\n'
                                             '        "100" : 0.023608,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 2,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring-bdev_uring: (groupid=0, '
                                             'jobs=1): err= 0: pid=7252: Wed '
                                             'Oct 26 15:09:59 2022\n'
                                             '  read: IOPS=259k, BW=126MiB/s '
                                             '(133MB/s)(885MiB/7001msec)\n'
                                             '    slat (nsec): min=110, '
                                             'max=12924, avg=173.49, '
                                             'stdev=32.37\n'
                                             '    clat (usec): min=5, max=118, '
                                             'avg= 7.28, stdev= 1.68\n'
                                             '     lat (usec): min=6, max=118, '
                                             'avg= 7.45, stdev= 1.68\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 6560],  '
                                             '5.00th=[ 6688], 10.00th=[ 6688], '
                                             '20.00th=[ 6752],\n'
                                             '     | 30.00th=[ 6752], '
                                             '40.00th=[ 6816], 50.00th=[ '
                                             '6880], 60.00th=[ 6880],\n'
                                             '     | 70.00th=[ 6944], '
                                             '80.00th=[ 7200], 90.00th=[ '
                                             '8256], 95.00th=[ 8384],\n'
                                             '     | 99.00th=[14656], '
                                             '99.50th=[18560], '
                                             '99.90th=[20352], '
                                             '99.95th=[21120],\n'
                                             '     | 99.99th=[62720]\n'
                                             '   bw (  KiB/s): min=128981, '
                                             'max=130259, per=100.00%, '
                                             'avg=129499.93, stdev=474.43, '
                                             'samples=14\n'
                                             '   iops        : min=257963, '
                                             'max=260518, avg=258999.93, '
                                             'stdev=948.78, samples=14\n'
                                             '  lat (usec)   : 10=97.06%, '
                                             '20=2.79%, 50=0.13%, 100=0.02%, '
                                             '250=0.01%\n'
                                             '  cpu          : usr=32.96%, '
                                             'sys=67.01%, ctx=11, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=8.4%, '
                                             '2=91.6%, 4=0.0%, 8=0.0%, '
                                             '16=0.0%, 32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=1812908,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=2\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=126MiB/s (133MB/s), '
                                             '126MiB/s-126MiB/s '
                                             '(133MB/s-133MB/s), io=885MiB '
                                             '(928MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_138.output'),
                                   'state': {'begin': 1666782709.2428133,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_2_io_uring-bdev_uring_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8369588851928711,
                                             'end': 1666782710.0797722,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_138.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_138.state')},
                       'cmd_139': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_512_2_io_uring-bdev_uring_1.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_139.output'),
                                   'state': {'begin': 1666782712.184676,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_2_io_uring-bdev_uring_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8070030212402344,
                                             'end': 1666782712.991679,
                                             'err': 1,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_139.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_139.state')},
                       'cmd_14': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_14.output'),
                                  'state': {'begin': 1666782102.530145,
                                            'cmd': '/opt/fio/bin/fio '
                                                   '--name="io_uring_cmd-xnvme" '
                                                   '--rw="randread" '
                                                   '--size="12G" --iodepth="1" '
                                                   '--bs="512" --direct="0" '
                                                   '--thread="1" '
                                                   '--time_based="1" '
                                                   '--runtime="7" '
                                                   '--ramp_time="3" '
                                                   '--norandommap="1" '
                                                   '--allow_file_create="0" '
                                                   '--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
                                                   '--xnvme_be="linux" '
                                                   '--xnvme_async="io_uring_cmd" '
                                                   '--xnvme_sync="nvme" '
                                                   '--xnvme_admin="nvme" '
                                                   '--xnvme_dev_nsid="1" '
                                                   '--filename="/dev/ng0n1" '
                                                   '--output-format="normal,json" '
                                                   '--output="/tmp/fio-output_512_1_io_uring_cmd-xnvme_1.txt" '
                                                   '--cpus_allowed="1"',
                                            'cwd': 'None',
                                            'elapsed': 11.075329542160034,
                                            'end': 1666782113.6054745,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_14.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_14.state')},
                       'cmd_140': {'output': 'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_140.output'),
                                   'state': {'begin': 1666782712.9987884,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring-bdev_uring" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="2" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="spdk_bdev" '
                                                    '--spdk_conf="/tmp/spdk_bdev_uring_foo.conf" '
                                                    '--filename="Nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_2_io_uring-bdev_uring_1.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.543930292129517,
                                             'end': 1666782724.5427186,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_140.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_140.state')},
                       'cmd_141': {'output': 'io_uring-bdev_uring: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=spdk_bdev, iodepth=2\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666789814,\n'
                                             '  "timestamp_ms" : '
                                             '1666789814565,\n'
                                             '  "time" : "Wed Oct 26 15:10:14 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring-bdev_uring",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring-bdev_uring",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "2",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"spdk_bdev",\n'
                                             '        "spdk_conf" : '
                                             '"/tmp/spdk_bdev_uring_foo.conf",\n'
                                             '        "filename" : "Nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : 919181824,\n'
                                             '        "io_kbytes" : 897638,\n'
                                             '        "bw_bytes" : 131292933,\n'
                                             '        "bw" : 128215,\n'
                                             '        "iops" : 256431.366948,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 1795276,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 110,\n'
                                             '          "max" : 11972,\n'
                                             '          "mean" : 171.000074,\n'
                                             '          "stddev" : 44.899254,\n'
                                             '          "N" : 1795276\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 5971,\n'
                                             '          "max" : 119064,\n'
                                             '          "mean" : 7345.391732,\n'
                                             '          "stddev" : '
                                             '1661.364011,\n'
                                             '          "N" : 1795277,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 6624,\n'
                                             '            "5.000000" : 6688,\n'
                                             '            "10.000000" : 6752,\n'
                                             '            "20.000000" : 6816,\n'
                                             '            "30.000000" : 6816,\n'
                                             '            "40.000000" : 6880,\n'
                                             '            "50.000000" : 6880,\n'
                                             '            "60.000000" : 6944,\n'
                                             '            "70.000000" : 7008,\n'
                                             '            "80.000000" : 7840,\n'
                                             '            "90.000000" : 8256,\n'
                                             '            "95.000000" : 8512,\n'
                                             '            "99.000000" : '
                                             '14784,\n'
                                             '            "99.500000" : '
                                             '18560,\n'
                                             '            "99.900000" : '
                                             '20096,\n'
                                             '            "99.950000" : '
                                             '20864,\n'
                                             '            "99.990000" : 62720\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 6151,\n'
                                             '          "max" : 119244,\n'
                                             '          "mean" : 7516.391811,\n'
                                             '          "stddev" : '
                                             '1658.455431,\n'
                                             '          "N" : 1795277\n'
                                             '        },\n'
                                             '        "bw_min" : 127602,\n'
                                             '        "bw_max" : 128753,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '128239.071429,\n'
                                             '        "bw_dev" : 367.154225,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 255204,\n'
                                             '        "iops_max" : 257506,\n'
                                             '        "iops_mean" : '
                                             '256478.214286,\n'
                                             '        "iops_stddev" : '
                                             '734.226771,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 33.300000,\n'
                                             '      "sys_cpu" : 66.671429,\n'
                                             '      "ctx" : 10,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 9.490184,\n'
                                             '        "2" : 90.509816,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 97.017896,\n'
                                             '        "20" : 2.866412,\n'
                                             '        "50" : 0.094359,\n'
                                             '        "100" : 0.021055,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 2,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring-bdev_uring: (groupid=0, '
                                             'jobs=1): err= 0: pid=7309: Wed '
                                             'Oct 26 15:10:14 2022\n'
                                             '  read: IOPS=256k, BW=125MiB/s '
                                             '(131MB/s)(877MiB/7001msec)\n'
                                             '    slat (nsec): min=110, '
                                             'max=11972, avg=171.00, '
                                             'stdev=44.90\n'
                                             '    clat (usec): min=5, max=119, '
                                             'avg= 7.35, stdev= 1.66\n'
                                             '     lat (usec): min=6, max=119, '
                                             'avg= 7.52, stdev= 1.66\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 6624],  '
                                             '5.00th=[ 6688], 10.00th=[ 6752], '
                                             '20.00th=[ 6816],\n'
                                             '     | 30.00th=[ 6816], '
                                             '40.00th=[ 6880], 50.00th=[ '
                                             '6880], 60.00th=[ 6944],\n'
                                             '     | 70.00th=[ 7008], '
                                             '80.00th=[ 7840], 90.00th=[ '
                                             '8256], 95.00th=[ 8512],\n'
                                             '     | 99.00th=[14784], '
                                             '99.50th=[18560], '
                                             '99.90th=[20096], '
                                             '99.95th=[20864],\n'
                                             '     | 99.99th=[62720]\n'
                                             '   bw (  KiB/s): min=127602, '
                                             'max=128753, per=100.00%, '
                                             'avg=128239.07, stdev=367.15, '
                                             'samples=14\n'
                                             '   iops        : min=255204, '
                                             'max=257506, avg=256478.21, '
                                             'stdev=734.23, samples=14\n'
                                             '  lat (usec)   : 10=97.02%, '
                                             '20=2.87%, 50=0.09%, 100=0.02%, '
                                             '250=0.01%\n'
                                             '  cpu          : usr=33.30%, '
                                             'sys=66.67%, ctx=10, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=9.5%, '
                                             '2=90.5%, 4=0.0%, 8=0.0%, '
                                             '16=0.0%, 32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=1795276,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=2\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=125MiB/s (131MB/s), '
                                             '125MiB/s-125MiB/s '
                                             '(131MB/s-131MB/s), io=877MiB '
                                             '(919MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_141.output'),
                                   'state': {'begin': 1666782724.5457072,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_2_io_uring-bdev_uring_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8194141387939453,
                                             'end': 1666782725.3651214,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_141.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_141.state')},
                       'cmd_142': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_512_2_io_uring-bdev_uring_2.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_142.output'),
                                   'state': {'begin': 1666782727.4584682,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_2_io_uring-bdev_uring_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7973694801330566,
                                             'end': 1666782728.2558377,
                                             'err': 1,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_142.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_142.state')},
                       'cmd_143': {'output': 'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_143.output'),
                                   'state': {'begin': 1666782728.2577453,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring-bdev_uring" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="2" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="spdk_bdev" '
                                                    '--spdk_conf="/tmp/spdk_bdev_uring_foo.conf" '
                                                    '--filename="Nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_2_io_uring-bdev_uring_2.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.487537860870361,
                                             'end': 1666782739.7452831,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_143.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_143.state')},
                       'cmd_144': {'output': 'io_uring-bdev_uring: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=spdk_bdev, iodepth=2\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666789829,\n'
                                             '  "timestamp_ms" : '
                                             '1666789829766,\n'
                                             '  "time" : "Wed Oct 26 15:10:29 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring-bdev_uring",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring-bdev_uring",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "2",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"spdk_bdev",\n'
                                             '        "spdk_conf" : '
                                             '"/tmp/spdk_bdev_uring_foo.conf",\n'
                                             '        "filename" : "Nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : 924888576,\n'
                                             '        "io_kbytes" : 903211,\n'
                                             '        "bw_bytes" : 132108066,\n'
                                             '        "bw" : 129011,\n'
                                             '        "iops" : 258023.425225,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 1806422,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 110,\n'
                                             '          "max" : 11772,\n'
                                             '          "mean" : 169.994528,\n'
                                             '          "stddev" : 41.933383,\n'
                                             '          "N" : 1806422\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 5952,\n'
                                             '          "max" : 102713,\n'
                                             '          "mean" : 7298.990791,\n'
                                             '          "stddev" : '
                                             '1647.972135,\n'
                                             '          "N" : 1806422,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 6624,\n'
                                             '            "5.000000" : 6688,\n'
                                             '            "10.000000" : 6752,\n'
                                             '            "20.000000" : 6752,\n'
                                             '            "30.000000" : 6816,\n'
                                             '            "40.000000" : 6816,\n'
                                             '            "50.000000" : 6880,\n'
                                             '            "60.000000" : 6880,\n'
                                             '            "70.000000" : 6944,\n'
                                             '            "80.000000" : 7264,\n'
                                             '            "90.000000" : 8256,\n'
                                             '            "95.000000" : 8384,\n'
                                             '            "99.000000" : '
                                             '14784,\n'
                                             '            "99.500000" : '
                                             '18560,\n'
                                             '            "99.900000" : '
                                             '20352,\n'
                                             '            "99.950000" : '
                                             '20864,\n'
                                             '            "99.990000" : 62208\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 6122,\n'
                                             '          "max" : 102883,\n'
                                             '          "mean" : 7468.985319,\n'
                                             '          "stddev" : '
                                             '1644.499905,\n'
                                             '          "N" : 1806422\n'
                                             '        },\n'
                                             '        "bw_min" : 125789,\n'
                                             '        "bw_max" : 129837,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '129033.714286,\n'
                                             '        "bw_dev" : 1030.428994,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 251579,\n'
                                             '        "iops_max" : 259674,\n'
                                             '        "iops_mean" : '
                                             '258067.500000,\n'
                                             '        "iops_stddev" : '
                                             '2060.615769,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 36.842857,\n'
                                             '      "sys_cpu" : 63.128571,\n'
                                             '      "ctx" : 16,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 8.782610,\n'
                                             '        "2" : 91.217390,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 97.043991,\n'
                                             '        "20" : 2.821212,\n'
                                             '        "50" : 0.114259,\n'
                                             '        "100" : 0.020427,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 2,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring-bdev_uring: (groupid=0, '
                                             'jobs=1): err= 0: pid=7367: Wed '
                                             'Oct 26 15:10:29 2022\n'
                                             '  read: IOPS=258k, BW=126MiB/s '
                                             '(132MB/s)(882MiB/7001msec)\n'
                                             '    slat (nsec): min=110, '
                                             'max=11772, avg=169.99, '
                                             'stdev=41.93\n'
                                             '    clat (usec): min=5, max=102, '
                                             'avg= 7.30, stdev= 1.65\n'
                                             '     lat (usec): min=6, max=102, '
                                             'avg= 7.47, stdev= 1.64\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 6624],  '
                                             '5.00th=[ 6688], 10.00th=[ 6752], '
                                             '20.00th=[ 6752],\n'
                                             '     | 30.00th=[ 6816], '
                                             '40.00th=[ 6816], 50.00th=[ '
                                             '6880], 60.00th=[ 6880],\n'
                                             '     | 70.00th=[ 6944], '
                                             '80.00th=[ 7264], 90.00th=[ '
                                             '8256], 95.00th=[ 8384],\n'
                                             '     | 99.00th=[14784], '
                                             '99.50th=[18560], '
                                             '99.90th=[20352], '
                                             '99.95th=[20864],\n'
                                             '     | 99.99th=[62208]\n'
                                             '   bw (  KiB/s): min=125789, '
                                             'max=129837, per=100.00%, '
                                             'avg=129033.71, stdev=1030.43, '
                                             'samples=14\n'
                                             '   iops        : min=251579, '
                                             'max=259674, avg=258067.50, '
                                             'stdev=2060.62, samples=14\n'
                                             '  lat (usec)   : 10=97.04%, '
                                             '20=2.82%, 50=0.11%, 100=0.02%, '
                                             '250=0.01%\n'
                                             '  cpu          : usr=36.84%, '
                                             'sys=63.13%, ctx=16, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=8.8%, '
                                             '2=91.2%, 4=0.0%, 8=0.0%, '
                                             '16=0.0%, 32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=1806422,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=2\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=126MiB/s (132MB/s), '
                                             '126MiB/s-126MiB/s '
                                             '(132MB/s-132MB/s), io=882MiB '
                                             '(925MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_144.output'),
                                   'state': {'begin': 1666782739.7478065,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_2_io_uring-bdev_uring_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8531312942504883,
                                             'end': 1666782740.6009378,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_144.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_144.state')},
                       'cmd_145': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_512_2_io_uring-bdev_xnvme_0.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_145.output'),
                                   'state': {'begin': 1666782742.8703396,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_2_io_uring-bdev_xnvme_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8204746246337891,
                                             'end': 1666782743.6908143,
                                             'err': 1,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_145.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_145.state')},
                       'cmd_146': {'output': 'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_146.output'),
                                   'state': {'begin': 1666782743.6930044,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring-bdev_xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="2" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="spdk_bdev" '
                                                    '--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" '
                                                    '--filename="Nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_2_io_uring-bdev_xnvme_0.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.546950101852417,
                                             'end': 1666782755.2399545,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_146.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_146.state')},
                       'cmd_147': {'output': 'io_uring-bdev_xnvme: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=spdk_bdev, iodepth=2\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666789845,\n'
                                             '  "timestamp_ms" : '
                                             '1666789845262,\n'
                                             '  "time" : "Wed Oct 26 15:10:45 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring-bdev_xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring-bdev_xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "2",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"spdk_bdev",\n'
                                             '        "spdk_conf" : '
                                             '"/tmp/spdk_bdev_xnvme_io_uring.conf",\n'
                                             '        "filename" : "Nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '1071219200,\n'
                                             '        "io_kbytes" : 1046112,\n'
                                             '        "bw_bytes" : 153009455,\n'
                                             '        "bw" : 149423,\n'
                                             '        "iops" : 298846.307670,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 2092223,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1683,\n'
                                             '          "max" : 35506,\n'
                                             '          "mean" : 1886.948066,\n'
                                             '          "stddev" : '
                                             '122.911174,\n'
                                             '          "N" : 2092224\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 260,\n'
                                             '          "max" : 119775,\n'
                                             '          "mean" : 4572.451873,\n'
                                             '          "stddev" : '
                                             '1788.671868,\n'
                                             '          "N" : 2092224,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 3888,\n'
                                             '            "5.000000" : 4128,\n'
                                             '            "10.000000" : 4192,\n'
                                             '            "20.000000" : 4256,\n'
                                             '            "30.000000" : 4256,\n'
                                             '            "40.000000" : 4320,\n'
                                             '            "50.000000" : 4320,\n'
                                             '            "60.000000" : 4320,\n'
                                             '            "70.000000" : 4384,\n'
                                             '            "80.000000" : 4448,\n'
                                             '            "90.000000" : 4448,\n'
                                             '            "95.000000" : 4640,\n'
                                             '            "99.000000" : '
                                             '12608,\n'
                                             '            "99.500000" : '
                                             '17536,\n'
                                             '            "99.900000" : '
                                             '21888,\n'
                                             '            "99.950000" : '
                                             '22144,\n'
                                             '            "99.990000" : 63232\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 5139,\n'
                                             '          "max" : 121608,\n'
                                             '          "mean" : 6459.399939,\n'
                                             '          "stddev" : '
                                             '1792.133017,\n'
                                             '          "N" : 2092224\n'
                                             '        },\n'
                                             '        "bw_min" : 148978,\n'
                                             '        "bw_max" : 149936,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '149450.785714,\n'
                                             '        "bw_dev" : 305.810695,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 297956,\n'
                                             '        "iops_max" : 299873,\n'
                                             '        "iops_mean" : '
                                             '298901.642857,\n'
                                             '        "iops_stddev" : '
                                             '612.123116,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 44.157143,\n'
                                             '      "sys_cpu" : 55.828571,\n'
                                             '      "ctx" : 13,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 100.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.010000,\n'
                                             '        "750" : 0.010000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 3.109707,\n'
                                             '        "10" : 94.906518,\n'
                                             '        "20" : 1.720562,\n'
                                             '        "50" : 0.241657,\n'
                                             '        "100" : 0.020982,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 2,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring-bdev_xnvme: (groupid=0, '
                                             'jobs=1): err= 0: pid=7424: Wed '
                                             'Oct 26 15:10:45 2022\n'
                                             '  read: IOPS=299k, BW=146MiB/s '
                                             '(153MB/s)(1022MiB/7001msec)\n'
                                             '    slat (nsec): min=1683, '
                                             'max=35506, avg=1886.95, '
                                             'stdev=122.91\n'
                                             '    clat (nsec): min=260, '
                                             'max=119775, avg=4572.45, '
                                             'stdev=1788.67\n'
                                             '     lat (usec): min=5, max=121, '
                                             'avg= 6.46, stdev= 1.79\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 3888],  '
                                             '5.00th=[ 4128], 10.00th=[ 4192], '
                                             '20.00th=[ 4256],\n'
                                             '     | 30.00th=[ 4256], '
                                             '40.00th=[ 4320], 50.00th=[ '
                                             '4320], 60.00th=[ 4320],\n'
                                             '     | 70.00th=[ 4384], '
                                             '80.00th=[ 4448], 90.00th=[ '
                                             '4448], 95.00th=[ 4640],\n'
                                             '     | 99.00th=[12608], '
                                             '99.50th=[17536], '
                                             '99.90th=[21888], '
                                             '99.95th=[22144],\n'
                                             '     | 99.99th=[63232]\n'
                                             '   bw (  KiB/s): min=148978, '
                                             'max=149936, per=100.00%, '
                                             'avg=149450.79, stdev=305.81, '
                                             'samples=14\n'
                                             '   iops        : min=297956, '
                                             'max=299873, avg=298901.64, '
                                             'stdev=612.12, samples=14\n'
                                             '  lat (nsec)   : 500=0.01%, '
                                             '750=0.01%\n'
                                             '  lat (usec)   : 4=3.11%, '
                                             '10=94.91%, 20=1.72%, 50=0.24%, '
                                             '100=0.02%\n'
                                             '  lat (usec)   : 250=0.01%\n'
                                             '  cpu          : usr=44.16%, '
                                             'sys=55.83%, ctx=13, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, '
                                             '2=100.0%, 4=0.0%, 8=0.0%, '
                                             '16=0.0%, 32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=2092223,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=2\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=146MiB/s (153MB/s), '
                                             '146MiB/s-146MiB/s '
                                             '(153MB/s-153MB/s), io=1022MiB '
                                             '(1071MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_147.output'),
                                   'state': {'begin': 1666782755.242614,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_2_io_uring-bdev_xnvme_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8503968715667725,
                                             'end': 1666782756.093011,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_147.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_147.state')},
                       'cmd_148': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_512_2_io_uring-bdev_xnvme_1.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_148.output'),
                                   'state': {'begin': 1666782758.268324,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_2_io_uring-bdev_xnvme_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8301882743835449,
                                             'end': 1666782759.0985122,
                                             'err': 1,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_148.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_148.state')},
                       'cmd_149': {'output': 'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_149.output'),
                                   'state': {'begin': 1666782759.1007652,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring-bdev_xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="2" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="spdk_bdev" '
                                                    '--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" '
                                                    '--filename="Nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_2_io_uring-bdev_xnvme_1.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.452395677566528,
                                             'end': 1666782770.553161,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_149.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_149.state')},
                       'cmd_15': {'output': 'io_uring_cmd-xnvme: (g=0): '
                                            'rw=randread, bs=(R) 512B-512B, '
                                            '(W) 512B-512B, (T) 512B-512B, '
                                            'ioengine=xnvme, iodepth=1\n'
                                            'fio-3.32-dirty\n'
                                            'Starting 1 thread\n'
                                            '{\n'
                                            '  "fio version" : '
                                            '"fio-3.32-dirty",\n'
                                            '  "timestamp" : 1666789203,\n'
                                            '  "timestamp_ms" : '
                                            '1666789203700,\n'
                                            '  "time" : "Wed Oct 26 15:00:03 '
                                            '2022",\n'
                                            '  "jobs" : [\n'
                                            '    {\n'
                                            '      "jobname" : '
                                            '"io_uring_cmd-xnvme",\n'
                                            '      "groupid" : 0,\n'
                                            '      "error" : 0,\n'
                                            '      "eta" : 0,\n'
                                            '      "elapsed" : 11,\n'
                                            '      "job options" : {\n'
                                            '        "name" : '
                                            '"io_uring_cmd-xnvme",\n'
                                            '        "rw" : "randread",\n'
                                            '        "size" : "12G",\n'
                                            '        "iodepth" : "1",\n'
                                            '        "bs" : "512",\n'
                                            '        "direct" : "0",\n'
                                            '        "thread" : "1",\n'
                                            '        "time_based" : "1",\n'
                                            '        "runtime" : "7",\n'
                                            '        "ramp_time" : "3",\n'
                                            '        "norandommap" : "1",\n'
                                            '        "allow_file_create" : '
                                            '"0",\n'
                                            '        "ioengine" : '
                                            '"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
                                            '        "xnvme_be" : "linux",\n'
                                            '        "xnvme_async" : '
                                            '"io_uring_cmd",\n'
                                            '        "xnvme_sync" : "nvme",\n'
                                            '        "xnvme_admin" : "nvme",\n'
                                            '        "xnvme_dev_nsid" : "1",\n'
                                            '        "filename" : '
                                            '"/dev/ng0n1",\n'
                                            '        "cpus_allowed" : "1"\n'
                                            '      },\n'
                                            '      "read" : {\n'
                                            '        "io_bytes" : 394408960,\n'
                                            '        "io_kbytes" : 385165,\n'
                                            '        "bw_bytes" : 56336089,\n'
                                            '        "bw" : 55015,\n'
                                            '        "iops" : 110031.424082,\n'
                                            '        "runtime" : 7001,\n'
                                            '        "total_ios" : 770330,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 1402,\n'
                                            '          "max" : 14557,\n'
                                            '          "mean" : 1512.964761,\n'
                                            '          "stddev" : 62.706972,\n'
                                            '          "N" : 770330\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 260,\n'
                                            '          "max" : 97473,\n'
                                            '          "mean" : 7280.894260,\n'
                                            '          "stddev" : 982.896349,\n'
                                            '          "N" : 770330,\n'
                                            '          "percentile" : {\n'
                                            '            "1.000000" : 6752,\n'
                                            '            "5.000000" : 7008,\n'
                                            '            "10.000000" : 7072,\n'
                                            '            "20.000000" : 7072,\n'
                                            '            "30.000000" : 7072,\n'
                                            '            "40.000000" : 7136,\n'
                                            '            "50.000000" : 7200,\n'
                                            '            "60.000000" : 7200,\n'
                                            '            "70.000000" : 7264,\n'
                                            '            "80.000000" : 7264,\n'
                                            '            "90.000000" : 7392,\n'
                                            '            "95.000000" : 7456,\n'
                                            '            "99.000000" : 10432,\n'
                                            '            "99.500000" : 13632,\n'
                                            '            "99.900000" : 15680,\n'
                                            '            "99.950000" : 15808,\n'
                                            '            "99.990000" : 58112\n'
                                            '          }\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 6993,\n'
                                            '          "max" : 98975,\n'
                                            '          "mean" : 8793.859020,\n'
                                            '          "stddev" : 985.772717,\n'
                                            '          "N" : 770330\n'
                                            '        },\n'
                                            '        "bw_min" : 54788,\n'
                                            '        "bw_max" : 55269,\n'
                                            '        "bw_agg" : 100.000000,\n'
                                            '        "bw_mean" : '
                                            '55025.500000,\n'
                                            '        "bw_dev" : 130.894408,\n'
                                            '        "bw_samples" : 14,\n'
                                            '        "iops_min" : 109576,\n'
                                            '        "iops_max" : 110538,\n'
                                            '        "iops_mean" : '
                                            '110051.000000,\n'
                                            '        "iops_stddev" : '
                                            '261.788817,\n'
                                            '        "iops_samples" : 14\n'
                                            '      },\n'
                                            '      "write" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "trim" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "sync" : {\n'
                                            '        "total_ios" : 0,\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        }\n'
                                            '      },\n'
                                            '      "job_runtime" : 7000,\n'
                                            '      "usr_cpu" : 8.414286,\n'
                                            '      "sys_cpu" : 35.785714,\n'
                                            '      "ctx" : 770332,\n'
                                            '      "majf" : 0,\n'
                                            '      "minf" : 0,\n'
                                            '      "iodepth_level" : {\n'
                                            '        "1" : 100.000000,\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_submit" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_complete" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ns" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.010000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_us" : {\n'
                                            '        "2" : 0.010000,\n'
                                            '        "4" : 0.010000,\n'
                                            '        "10" : 98.690691,\n'
                                            '        "20" : 1.290486,\n'
                                            '        "50" : 0.010000,\n'
                                            '        "100" : 0.014799,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ms" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000,\n'
                                            '        "2000" : 0.000000,\n'
                                            '        ">=2000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_depth" : 1,\n'
                                            '      "latency_target" : 0,\n'
                                            '      "latency_percentile" : '
                                            '100.000000,\n'
                                            '      "latency_window" : 0\n'
                                            '    }\n'
                                            '  ]\n'
                                            '}\n'
                                            '\n'
                                            'io_uring_cmd-xnvme: (groupid=0, '
                                            'jobs=1): err= 0: pid=5122: Wed '
                                            'Oct 26 15:00:03 2022\n'
                                            '  read: IOPS=110k, BW=53.7MiB/s '
                                            '(56.3MB/s)(376MiB/7001msec)\n'
                                            '    slat (nsec): min=1402, '
                                            'max=14557, avg=1512.96, '
                                            'stdev=62.71\n'
                                            '    clat (nsec): min=260, '
                                            'max=97473, avg=7280.89, '
                                            'stdev=982.90\n'
                                            '     lat (nsec): min=6993, '
                                            'max=98975, avg=8793.86, '
                                            'stdev=985.77\n'
                                            '    clat percentiles (nsec):\n'
                                            '     |  1.00th=[ 6752],  5.00th=[ '
                                            '7008], 10.00th=[ 7072], 20.00th=[ '
                                            '7072],\n'
                                            '     | 30.00th=[ 7072], 40.00th=[ '
                                            '7136], 50.00th=[ 7200], 60.00th=[ '
                                            '7200],\n'
                                            '     | 70.00th=[ 7264], 80.00th=[ '
                                            '7264], 90.00th=[ 7392], 95.00th=[ '
                                            '7456],\n'
                                            '     | 99.00th=[10432], '
                                            '99.50th=[13632], 99.90th=[15680], '
                                            '99.95th=[15808],\n'
                                            '     | 99.99th=[58112]\n'
                                            '   bw (  KiB/s): min=54788, '
                                            'max=55269, per=100.00%, '
                                            'avg=55025.50, stdev=130.89, '
                                            'samples=14\n'
                                            '   iops        : min=109576, '
                                            'max=110538, avg=110051.00, '
                                            'stdev=261.79, samples=14\n'
                                            '  lat (nsec)   : 500=0.01%\n'
                                            '  lat (usec)   : 2=0.01%, '
                                            '4=0.01%, 10=98.69%, 20=1.29%, '
                                            '50=0.01%\n'
                                            '  lat (usec)   : 100=0.01%\n'
                                            '  cpu          : usr=8.41%, '
                                            'sys=35.79%, ctx=770332, majf=0, '
                                            'minf=0\n'
                                            '  IO depths    : 1=100.0%, '
                                            '2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, >=64=0.0%\n'
                                            '     submit    : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     complete  : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     issued rwts: '
                                            'total=770330,0,0,0 short=0,0,0,0 '
                                            'dropped=0,0,0,0\n'
                                            '     latency   : target=0, '
                                            'window=0, percentile=100.00%, '
                                            'depth=1\n'
                                            '\n'
                                            'Run status group 0 (all jobs):\n'
                                            '   READ: bw=53.7MiB/s (56.3MB/s), '
                                            '53.7MiB/s-53.7MiB/s '
                                            '(56.3MB/s-56.3MB/s), io=376MiB '
                                            '(394MB), run=7001-7001msec\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_15.output'),
                                  'state': {'begin': 1666782113.6086535,
                                            'cmd': 'cat '
                                                   '/tmp/fio-output_512_1_io_uring_cmd-xnvme_1.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.8623161315917969,
                                            'end': 1666782114.4709697,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_15.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_15.state')},
                       'cmd_150': {'output': 'io_uring-bdev_xnvme: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=spdk_bdev, iodepth=2\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666789860,\n'
                                             '  "timestamp_ms" : '
                                             '1666789860552,\n'
                                             '  "time" : "Wed Oct 26 15:11:00 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring-bdev_xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring-bdev_xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "2",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"spdk_bdev",\n'
                                             '        "spdk_conf" : '
                                             '"/tmp/spdk_bdev_xnvme_io_uring.conf",\n'
                                             '        "filename" : "Nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '1074551808,\n'
                                             '        "io_kbytes" : 1049367,\n'
                                             '        "bw_bytes" : 153485474,\n'
                                             '        "bw" : 149888,\n'
                                             '        "iops" : 299776.174832,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 2098733,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1673,\n'
                                             '          "max" : 23223,\n'
                                             '          "mean" : 1867.469730,\n'
                                             '          "stddev" : '
                                             '122.869433,\n'
                                             '          "N" : 2098733\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 230,\n'
                                             '          "max" : 115426,\n'
                                             '          "mean" : 4579.960121,\n'
                                             '          "stddev" : '
                                             '1762.455468,\n'
                                             '          "N" : 2098734,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 3888,\n'
                                             '            "5.000000" : 4128,\n'
                                             '            "10.000000" : 4192,\n'
                                             '            "20.000000" : 4256,\n'
                                             '            "30.000000" : 4256,\n'
                                             '            "40.000000" : 4320,\n'
                                             '            "50.000000" : 4320,\n'
                                             '            "60.000000" : 4384,\n'
                                             '            "70.000000" : 4384,\n'
                                             '            "80.000000" : 4448,\n'
                                             '            "90.000000" : 4512,\n'
                                             '            "95.000000" : 4640,\n'
                                             '            "99.000000" : '
                                             '12608,\n'
                                             '            "99.500000" : '
                                             '17536,\n'
                                             '            "99.900000" : '
                                             '21888,\n'
                                             '            "99.950000" : '
                                             '22400,\n'
                                             '            "99.990000" : 61184\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 5070,\n'
                                             '          "max" : 117269,\n'
                                             '          "mean" : 6447.429815,\n'
                                             '          "stddev" : '
                                             '1765.832177,\n'
                                             '          "N" : 2098734\n'
                                             '        },\n'
                                             '        "bw_min" : 149368,\n'
                                             '        "bw_max" : 150878,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '149916.857143,\n'
                                             '        "bw_dev" : 485.029080,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 298736,\n'
                                             '        "iops_max" : 301757,\n'
                                             '        "iops_mean" : '
                                             '299833.785714,\n'
                                             '        "iops_stddev" : '
                                             '970.210618,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 28.900000,\n'
                                             '      "sys_cpu" : 71.085714,\n'
                                             '      "ctx" : 9,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 100.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.010000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.010000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.010000,\n'
                                             '        "4" : 3.019584,\n'
                                             '        "10" : 94.994837,\n'
                                             '        "20" : 1.723421,\n'
                                             '        "50" : 0.241717,\n'
                                             '        "100" : 0.019631,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 2,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring-bdev_xnvme: (groupid=0, '
                                             'jobs=1): err= 0: pid=7482: Wed '
                                             'Oct 26 15:11:00 2022\n'
                                             '  read: IOPS=300k, BW=146MiB/s '
                                             '(153MB/s)(1025MiB/7001msec)\n'
                                             '    slat (nsec): min=1673, '
                                             'max=23223, avg=1867.47, '
                                             'stdev=122.87\n'
                                             '    clat (nsec): min=230, '
                                             'max=115426, avg=4579.96, '
                                             'stdev=1762.46\n'
                                             '     lat (usec): min=5, max=117, '
                                             'avg= 6.45, stdev= 1.77\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 3888],  '
                                             '5.00th=[ 4128], 10.00th=[ 4192], '
                                             '20.00th=[ 4256],\n'
                                             '     | 30.00th=[ 4256], '
                                             '40.00th=[ 4320], 50.00th=[ '
                                             '4320], 60.00th=[ 4384],\n'
                                             '     | 70.00th=[ 4384], '
                                             '80.00th=[ 4448], 90.00th=[ '
                                             '4512], 95.00th=[ 4640],\n'
                                             '     | 99.00th=[12608], '
                                             '99.50th=[17536], '
                                             '99.90th=[21888], '
                                             '99.95th=[22400],\n'
                                             '     | 99.99th=[61184]\n'
                                             '   bw (  KiB/s): min=149368, '
                                             'max=150878, per=100.00%, '
                                             'avg=149916.86, stdev=485.03, '
                                             'samples=14\n'
                                             '   iops        : min=298736, '
                                             'max=301757, avg=299833.79, '
                                             'stdev=970.21, samples=14\n'
                                             '  lat (nsec)   : 250=0.01%, '
                                             '500=0.01%, 1000=0.01%\n'
                                             '  lat (usec)   : 2=0.01%, '
                                             '4=3.02%, 10=94.99%, 20=1.72%, '
                                             '50=0.24%\n'
                                             '  lat (usec)   : 100=0.02%, '
                                             '250=0.01%\n'
                                             '  cpu          : usr=28.90%, '
                                             'sys=71.09%, ctx=9, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, '
                                             '2=100.0%, 4=0.0%, 8=0.0%, '
                                             '16=0.0%, 32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=2098733,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=2\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=146MiB/s (153MB/s), '
                                             '146MiB/s-146MiB/s '
                                             '(153MB/s-153MB/s), io=1025MiB '
                                             '(1075MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_150.output'),
                                   'state': {'begin': 1666782770.5560105,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_2_io_uring-bdev_xnvme_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.801260232925415,
                                             'end': 1666782771.3572707,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_150.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_150.state')},
                       'cmd_151': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_512_2_io_uring-bdev_xnvme_2.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_151.output'),
                                   'state': {'begin': 1666782773.4524174,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_2_io_uring-bdev_xnvme_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7857146263122559,
                                             'end': 1666782774.238132,
                                             'err': 1,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_151.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_151.state')},
                       'cmd_152': {'output': 'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_152.output'),
                                   'state': {'begin': 1666782774.2401013,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring-bdev_xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="2" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="spdk_bdev" '
                                                    '--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" '
                                                    '--filename="Nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_2_io_uring-bdev_xnvme_2.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.518940925598145,
                                             'end': 1666782785.7590423,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_152.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_152.state')},
                       'cmd_153': {'output': 'io_uring-bdev_xnvme: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=spdk_bdev, iodepth=2\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666789875,\n'
                                             '  "timestamp_ms" : '
                                             '1666789875781,\n'
                                             '  "time" : "Wed Oct 26 15:11:15 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring-bdev_xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring-bdev_xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "2",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"spdk_bdev",\n'
                                             '        "spdk_conf" : '
                                             '"/tmp/spdk_bdev_xnvme_io_uring.conf",\n'
                                             '        "filename" : "Nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '1068184064,\n'
                                             '        "io_kbytes" : 1043148,\n'
                                             '        "bw_bytes" : 152575926,\n'
                                             '        "bw" : 148999,\n'
                                             '        "iops" : 297999.714327,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 2086296,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1723,\n'
                                             '          "max" : 23454,\n'
                                             '          "mean" : 1919.903935,\n'
                                             '          "stddev" : '
                                             '128.714954,\n'
                                             '          "N" : 2086296\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 250,\n'
                                             '          "max" : 98895,\n'
                                             '          "mean" : 4553.953938,\n'
                                             '          "stddev" : '
                                             '1784.346449,\n'
                                             '          "N" : 2086297,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 3856,\n'
                                             '            "5.000000" : 4128,\n'
                                             '            "10.000000" : 4192,\n'
                                             '            "20.000000" : 4192,\n'
                                             '            "30.000000" : 4256,\n'
                                             '            "40.000000" : 4256,\n'
                                             '            "50.000000" : 4320,\n'
                                             '            "60.000000" : 4320,\n'
                                             '            "70.000000" : 4384,\n'
                                             '            "80.000000" : 4384,\n'
                                             '            "90.000000" : 4448,\n'
                                             '            "95.000000" : 4640,\n'
                                             '            "99.000000" : '
                                             '12608,\n'
                                             '            "99.500000" : '
                                             '17536,\n'
                                             '            "99.900000" : '
                                             '21888,\n'
                                             '            "99.950000" : '
                                             '22144,\n'
                                             '            "99.990000" : 62720\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 5169,\n'
                                             '          "max" : 100759,\n'
                                             '          "mean" : 6473.857861,\n'
                                             '          "stddev" : '
                                             '1787.577180,\n'
                                             '          "N" : 2086297\n'
                                             '        },\n'
                                             '        "bw_min" : 148578,\n'
                                             '        "bw_max" : 149622,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '149027.000000,\n'
                                             '        "bw_dev" : 264.555069,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 297156,\n'
                                             '        "iops_max" : 299244,\n'
                                             '        "iops_mean" : '
                                             '298054.000000,\n'
                                             '        "iops_stddev" : '
                                             '529.110138,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 48.900000,\n'
                                             '      "sys_cpu" : 51.085714,\n'
                                             '      "ctx" : 15,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 100.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.010000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 3.277243,\n'
                                             '        "10" : 94.723520,\n'
                                             '        "20" : 1.731921,\n'
                                             '        "50" : 0.245459,\n'
                                             '        "100" : 0.021665,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 2,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring-bdev_xnvme: (groupid=0, '
                                             'jobs=1): err= 0: pid=7539: Wed '
                                             'Oct 26 15:11:15 2022\n'
                                             '  read: IOPS=298k, BW=146MiB/s '
                                             '(153MB/s)(1019MiB/7001msec)\n'
                                             '    slat (nsec): min=1723, '
                                             'max=23454, avg=1919.90, '
                                             'stdev=128.71\n'
                                             '    clat (nsec): min=250, '
                                             'max=98895, avg=4553.95, '
                                             'stdev=1784.35\n'
                                             '     lat (usec): min=5, max=100, '
                                             'avg= 6.47, stdev= 1.79\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 3856],  '
                                             '5.00th=[ 4128], 10.00th=[ 4192], '
                                             '20.00th=[ 4192],\n'
                                             '     | 30.00th=[ 4256], '
                                             '40.00th=[ 4256], 50.00th=[ '
                                             '4320], 60.00th=[ 4320],\n'
                                             '     | 70.00th=[ 4384], '
                                             '80.00th=[ 4384], 90.00th=[ '
                                             '4448], 95.00th=[ 4640],\n'
                                             '     | 99.00th=[12608], '
                                             '99.50th=[17536], '
                                             '99.90th=[21888], '
                                             '99.95th=[22144],\n'
                                             '     | 99.99th=[62720]\n'
                                             '   bw (  KiB/s): min=148578, '
                                             'max=149622, per=100.00%, '
                                             'avg=149027.00, stdev=264.56, '
                                             'samples=14\n'
                                             '   iops        : min=297156, '
                                             'max=299244, avg=298054.00, '
                                             'stdev=529.11, samples=14\n'
                                             '  lat (nsec)   : 500=0.01%\n'
                                             '  lat (usec)   : 4=3.28%, '
                                             '10=94.72%, 20=1.73%, 50=0.25%, '
                                             '100=0.02%\n'
                                             '  cpu          : usr=48.90%, '
                                             'sys=51.09%, ctx=15, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, '
                                             '2=100.0%, 4=0.0%, 8=0.0%, '
                                             '16=0.0%, 32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=2086296,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=2\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=146MiB/s (153MB/s), '
                                             '146MiB/s-146MiB/s '
                                             '(153MB/s-153MB/s), io=1019MiB '
                                             '(1068MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_153.output'),
                                   'state': {'begin': 1666782785.7616816,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_2_io_uring-bdev_xnvme_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8182222843170166,
                                             'end': 1666782786.5799038,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_153.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_153.state')},
                       'cmd_154': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_512_2_io_uring-xnvme_0.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_154.output'),
                                   'state': {'begin': 1666782787.6278014,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_2_io_uring-xnvme_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8002474308013916,
                                             'end': 1666782788.4280488,
                                             'err': 1,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_154.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_154.state')},
                       'cmd_155': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_155.output'),
                                   'state': {'begin': 1666782788.430456,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring-xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="2" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
                                                    '--xnvme_be="linux" '
                                                    '--xnvme_async="io_uring" '
                                                    '--xnvme_sync="nvme" '
                                                    '--xnvme_admin="nvme" '
                                                    '--xnvme_dev_nsid="1" '
                                                    '--filename="/dev/nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_2_io_uring-xnvme_0.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.187855958938599,
                                             'end': 1666782799.618312,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_155.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_155.state')},
                       'cmd_156': {'output': 'io_uring-xnvme: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=xnvme, iodepth=2\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666789889,\n'
                                             '  "timestamp_ms" : '
                                             '1666789889671,\n'
                                             '  "time" : "Wed Oct 26 15:11:29 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring-xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring-xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "2",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
                                             '        "xnvme_be" : "linux",\n'
                                             '        "xnvme_async" : '
                                             '"io_uring",\n'
                                             '        "xnvme_sync" : "nvme",\n'
                                             '        "xnvme_admin" : "nvme",\n'
                                             '        "xnvme_dev_nsid" : "1",\n'
                                             '        "filename" : '
                                             '"/dev/nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : 755842560,\n'
                                             '        "io_kbytes" : 738127,\n'
                                             '        "bw_bytes" : 107962085,\n'
                                             '        "bw" : 105431,\n'
                                             '        "iops" : 210863.305242,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 1476254,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1373,\n'
                                             '          "max" : 14708,\n'
                                             '          "mean" : 1626.049088,\n'
                                             '          "stddev" : '
                                             '183.510630,\n'
                                             '          "N" : 1476254\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 180,\n'
                                             '          "max" : 118923,\n'
                                             '          "mean" : 7596.831394,\n'
                                             '          "stddev" : '
                                             '2223.151074,\n'
                                             '          "N" : 1476255,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 5152,\n'
                                             '            "5.000000" : 5280,\n'
                                             '            "10.000000" : 5344,\n'
                                             '            "20.000000" : 5472,\n'
                                             '            "30.000000" : 6752,\n'
                                             '            "40.000000" : 7264,\n'
                                             '            "50.000000" : 7456,\n'
                                             '            "60.000000" : 7712,\n'
                                             '            "70.000000" : 8032,\n'
                                             '            "80.000000" : 9280,\n'
                                             '            "90.000000" : 9408,\n'
                                             '            "95.000000" : 9536,\n'
                                             '            "99.000000" : '
                                             '16768,\n'
                                             '            "99.500000" : '
                                             '18560,\n'
                                             '            "99.900000" : '
                                             '23936,\n'
                                             '            "99.950000" : '
                                             '24448,\n'
                                             '            "99.990000" : 61184\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 5811,\n'
                                             '          "max" : 120425,\n'
                                             '          "mean" : 9222.880527,\n'
                                             '          "stddev" : '
                                             '2251.499860,\n'
                                             '          "N" : 1476255\n'
                                             '        },\n'
                                             '        "bw_min" : 105209,\n'
                                             '        "bw_max" : 105680,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '105450.357143,\n'
                                             '        "bw_dev" : 127.924020,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 210418,\n'
                                             '        "iops_max" : 211362,\n'
                                             '        "iops_mean" : '
                                             '210900.785714,\n'
                                             '        "iops_stddev" : '
                                             '255.828722,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 14.514286,\n'
                                             '      "sys_cpu" : 51.542857,\n'
                                             '      "ctx" : 741997,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 100.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 97.311100,\n'
                                             '        "20" : 2.439621,\n'
                                             '        "50" : 0.215004,\n'
                                             '        "100" : 0.024522,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 2,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring-xnvme: (groupid=0, '
                                             'jobs=1): err= 0: pid=7587: Wed '
                                             'Oct 26 15:11:29 2022\n'
                                             '  read: IOPS=211k, BW=103MiB/s '
                                             '(108MB/s)(721MiB/7001msec)\n'
                                             '    slat (nsec): min=1373, '
                                             'max=14708, avg=1626.05, '
                                             'stdev=183.51\n'
                                             '    clat (nsec): min=180, '
                                             'max=118923, avg=7596.83, '
                                             'stdev=2223.15\n'
                                             '     lat (usec): min=5, max=120, '
                                             'avg= 9.22, stdev= 2.25\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 5152],  '
                                             '5.00th=[ 5280], 10.00th=[ 5344], '
                                             '20.00th=[ 5472],\n'
                                             '     | 30.00th=[ 6752], '
                                             '40.00th=[ 7264], 50.00th=[ '
                                             '7456], 60.00th=[ 7712],\n'
                                             '     | 70.00th=[ 8032], '
                                             '80.00th=[ 9280], 90.00th=[ '
                                             '9408], 95.00th=[ 9536],\n'
                                             '     | 99.00th=[16768], '
                                             '99.50th=[18560], '
                                             '99.90th=[23936], '
                                             '99.95th=[24448],\n'
                                             '     | 99.99th=[61184]\n'
                                             '   bw (  KiB/s): min=105209, '
                                             'max=105680, per=100.00%, '
                                             'avg=105450.36, stdev=127.92, '
                                             'samples=14\n'
                                             '   iops        : min=210418, '
                                             'max=211362, avg=210900.79, '
                                             'stdev=255.83, samples=14\n'
                                             '  lat (nsec)   : 250=0.01%\n'
                                             '  lat (usec)   : 4=0.01%, '
                                             '10=97.31%, 20=2.44%, 50=0.22%, '
                                             '100=0.02%\n'
                                             '  lat (usec)   : 250=0.01%\n'
                                             '  cpu          : usr=14.51%, '
                                             'sys=51.54%, ctx=741997, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, '
                                             '2=100.0%, 4=0.0%, 8=0.0%, '
                                             '16=0.0%, 32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=1476254,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=2\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=103MiB/s (108MB/s), '
                                             '103MiB/s-103MiB/s '
                                             '(108MB/s-108MB/s), io=721MiB '
                                             '(756MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_156.output'),
                                   'state': {'begin': 1666782799.6249485,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_2_io_uring-xnvme_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.805732011795044,
                                             'end': 1666782800.4306805,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_156.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_156.state')},
                       'cmd_157': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_512_2_io_uring-xnvme_1.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_157.output'),
                                   'state': {'begin': 1666782801.4764647,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_2_io_uring-xnvme_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7939236164093018,
                                             'end': 1666782802.2703884,
                                             'err': 1,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_157.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_157.state')},
                       'cmd_158': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_158.output'),
                                   'state': {'begin': 1666782802.2725344,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring-xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="2" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
                                                    '--xnvme_be="linux" '
                                                    '--xnvme_async="io_uring" '
                                                    '--xnvme_sync="nvme" '
                                                    '--xnvme_admin="nvme" '
                                                    '--xnvme_dev_nsid="1" '
                                                    '--filename="/dev/nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_2_io_uring-xnvme_1.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.128751754760742,
                                             'end': 1666782813.4012861,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_158.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_158.state')},
                       'cmd_159': {'output': 'io_uring-xnvme: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=xnvme, iodepth=2\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666789903,\n'
                                             '  "timestamp_ms" : '
                                             '1666789903447,\n'
                                             '  "time" : "Wed Oct 26 15:11:43 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring-xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring-xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "2",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
                                             '        "xnvme_be" : "linux",\n'
                                             '        "xnvme_async" : '
                                             '"io_uring",\n'
                                             '        "xnvme_sync" : "nvme",\n'
                                             '        "xnvme_admin" : "nvme",\n'
                                             '        "xnvme_dev_nsid" : "1",\n'
                                             '        "filename" : '
                                             '"/dev/nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : 751161856,\n'
                                             '        "io_kbytes" : 733556,\n'
                                             '        "bw_bytes" : 107293508,\n'
                                             '        "bw" : 104778,\n'
                                             '        "iops" : 209557.491787,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 1467112,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1402,\n'
                                             '          "max" : 30337,\n'
                                             '          "mean" : 1675.573352,\n'
                                             '          "stddev" : '
                                             '230.937066,\n'
                                             '          "N" : 1467112\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 2745,\n'
                                             '          "max" : 119153,\n'
                                             '          "mean" : 7591.607664,\n'
                                             '          "stddev" : '
                                             '2285.968158,\n'
                                             '          "N" : 1467113,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 5216,\n'
                                             '            "5.000000" : 5280,\n'
                                             '            "10.000000" : 5344,\n'
                                             '            "20.000000" : 5472,\n'
                                             '            "30.000000" : 5984,\n'
                                             '            "40.000000" : 7136,\n'
                                             '            "50.000000" : 7456,\n'
                                             '            "60.000000" : 7712,\n'
                                             '            "70.000000" : 8160,\n'
                                             '            "80.000000" : 9408,\n'
                                             '            "90.000000" : 9664,\n'
                                             '            "95.000000" : 9792,\n'
                                             '            "99.000000" : '
                                             '16768,\n'
                                             '            "99.500000" : '
                                             '18560,\n'
                                             '            "99.900000" : '
                                             '24192,\n'
                                             '            "99.950000" : '
                                             '24448,\n'
                                             '            "99.990000" : 64256\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 5801,\n'
                                             '          "max" : 120656,\n'
                                             '          "mean" : 9267.180865,\n'
                                             '          "stddev" : '
                                             '2311.110610,\n'
                                             '          "N" : 1467113\n'
                                             '        },\n'
                                             '        "bw_min" : 104352,\n'
                                             '        "bw_max" : 105176,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '104797.642857,\n'
                                             '        "bw_dev" : 211.042689,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 208705,\n'
                                             '        "iops_max" : 210352,\n'
                                             '        "iops_mean" : '
                                             '209595.357143,\n'
                                             '        "iops_stddev" : '
                                             '421.922998,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 15.942857,\n'
                                             '      "sys_cpu" : 51.414286,\n'
                                             '      "ctx" : 737441,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 100.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.010156,\n'
                                             '        "10" : 97.018701,\n'
                                             '        "20" : 2.717039,\n'
                                             '        "50" : 0.231816,\n'
                                             '        "100" : 0.022016,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 2,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring-xnvme: (groupid=0, '
                                             'jobs=1): err= 0: pid=7635: Wed '
                                             'Oct 26 15:11:43 2022\n'
                                             '  read: IOPS=210k, BW=102MiB/s '
                                             '(107MB/s)(716MiB/7001msec)\n'
                                             '    slat (nsec): min=1402, '
                                             'max=30337, avg=1675.57, '
                                             'stdev=230.94\n'
                                             '    clat (usec): min=2, max=119, '
                                             'avg= 7.59, stdev= 2.29\n'
                                             '     lat (usec): min=5, max=120, '
                                             'avg= 9.27, stdev= 2.31\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 5216],  '
                                             '5.00th=[ 5280], 10.00th=[ 5344], '
                                             '20.00th=[ 5472],\n'
                                             '     | 30.00th=[ 5984], '
                                             '40.00th=[ 7136], 50.00th=[ '
                                             '7456], 60.00th=[ 7712],\n'
                                             '     | 70.00th=[ 8160], '
                                             '80.00th=[ 9408], 90.00th=[ '
                                             '9664], 95.00th=[ 9792],\n'
                                             '     | 99.00th=[16768], '
                                             '99.50th=[18560], '
                                             '99.90th=[24192], '
                                             '99.95th=[24448],\n'
                                             '     | 99.99th=[64256]\n'
                                             '   bw (  KiB/s): min=104352, '
                                             'max=105176, per=100.00%, '
                                             'avg=104797.64, stdev=211.04, '
                                             'samples=14\n'
                                             '   iops        : min=208705, '
                                             'max=210352, avg=209595.36, '
                                             'stdev=421.92, samples=14\n'
                                             '  lat (usec)   : 4=0.01%, '
                                             '10=97.02%, 20=2.72%, 50=0.23%, '
                                             '100=0.02%\n'
                                             '  lat (usec)   : 250=0.01%\n'
                                             '  cpu          : usr=15.94%, '
                                             'sys=51.41%, ctx=737441, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, '
                                             '2=100.0%, 4=0.0%, 8=0.0%, '
                                             '16=0.0%, 32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=1467112,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=2\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=102MiB/s (107MB/s), '
                                             '102MiB/s-102MiB/s '
                                             '(107MB/s-107MB/s), io=716MiB '
                                             '(751MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_159.output'),
                                   'state': {'begin': 1666782813.4044902,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_2_io_uring-xnvme_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8058393001556396,
                                             'end': 1666782814.2103295,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_159.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_159.state')},
                       'cmd_16': {'output': 'rm: cannot remove '
                                            "'/tmp/fio-output_512_1_io_uring_cmd-xnvme_2.txt': "
                                            'No such file or directory\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_16.output'),
                                  'state': {'begin': 1666782115.5445716,
                                            'cmd': 'rm '
                                                   '/tmp/fio-output_512_1_io_uring_cmd-xnvme_2.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.8334617614746094,
                                            'end': 1666782116.3780334,
                                            'err': 1,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_16.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_16.state')},
                       'cmd_160': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_512_2_io_uring-xnvme_2.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_160.output'),
                                   'state': {'begin': 1666782815.1958127,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_2_io_uring-xnvme_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8335492610931396,
                                             'end': 1666782816.029362,
                                             'err': 1,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_160.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_160.state')},
                       'cmd_161': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_161.output'),
                                   'state': {'begin': 1666782816.03119,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring-xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="2" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
                                                    '--xnvme_be="linux" '
                                                    '--xnvme_async="io_uring" '
                                                    '--xnvme_sync="nvme" '
                                                    '--xnvme_admin="nvme" '
                                                    '--xnvme_dev_nsid="1" '
                                                    '--filename="/dev/nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_2_io_uring-xnvme_2.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.07723879814148,
                                             'end': 1666782827.1084287,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_161.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_161.state')},
                       'cmd_162': {'output': 'io_uring-xnvme: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=xnvme, iodepth=2\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666789917,\n'
                                             '  "timestamp_ms" : '
                                             '1666789917192,\n'
                                             '  "time" : "Wed Oct 26 15:11:57 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring-xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring-xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "2",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
                                             '        "xnvme_be" : "linux",\n'
                                             '        "xnvme_async" : '
                                             '"io_uring",\n'
                                             '        "xnvme_sync" : "nvme",\n'
                                             '        "xnvme_admin" : "nvme",\n'
                                             '        "xnvme_dev_nsid" : "1",\n'
                                             '        "filename" : '
                                             '"/dev/nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : 754146304,\n'
                                             '        "io_kbytes" : 736471,\n'
                                             '        "bw_bytes" : 107735186,\n'
                                             '        "bw" : 105210,\n'
                                             '        "iops" : 210420.142857,\n'
                                             '        "runtime" : 7000,\n'
                                             '        "total_ios" : 1472941,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1373,\n'
                                             '          "max" : 39404,\n'
                                             '          "mean" : 1621.156412,\n'
                                             '          "stddev" : '
                                             '192.150453,\n'
                                             '          "N" : 1472941\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 180,\n'
                                             '          "max" : 110587,\n'
                                             '          "mean" : 7626.157111,\n'
                                             '          "stddev" : '
                                             '2228.400265,\n'
                                             '          "N" : 1472942,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 5216,\n'
                                             '            "5.000000" : 5280,\n'
                                             '            "10.000000" : 5344,\n'
                                             '            "20.000000" : 5472,\n'
                                             '            "30.000000" : 6752,\n'
                                             '            "40.000000" : 7264,\n'
                                             '            "50.000000" : 7456,\n'
                                             '            "60.000000" : 7712,\n'
                                             '            "70.000000" : 8096,\n'
                                             '            "80.000000" : 9280,\n'
                                             '            "90.000000" : 9408,\n'
                                             '            "95.000000" : 9664,\n'
                                             '            "99.000000" : '
                                             '16768,\n'
                                             '            "99.500000" : '
                                             '18560,\n'
                                             '            "99.900000" : '
                                             '23936,\n'
                                             '            "99.950000" : '
                                             '24448,\n'
                                             '            "99.990000" : 61696\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 5741,\n'
                                             '          "max" : 112090,\n'
                                             '          "mean" : 9247.313388,\n'
                                             '          "stddev" : '
                                             '2260.379617,\n'
                                             '          "N" : 1472942\n'
                                             '        },\n'
                                             '        "bw_min" : 104853,\n'
                                             '        "bw_max" : 105510,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '105237.769231,\n'
                                             '        "bw_dev" : 207.918716,\n'
                                             '        "bw_samples" : 13,\n'
                                             '        "iops_min" : 209706,\n'
                                             '        "iops_max" : 211020,\n'
                                             '        "iops_mean" : '
                                             '210475.769231,\n'
                                             '        "iops_stddev" : '
                                             '415.849162,\n'
                                             '        "iops_samples" : 13\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 14.157143,\n'
                                             '      "sys_cpu" : 51.885714,\n'
                                             '      "ctx" : 740379,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 100.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 97.277556,\n'
                                             '        "20" : 2.467716,\n'
                                             '        "50" : 0.220783,\n'
                                             '        "100" : 0.024101,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 2,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring-xnvme: (groupid=0, '
                                             'jobs=1): err= 0: pid=7683: Wed '
                                             'Oct 26 15:11:57 2022\n'
                                             '  read: IOPS=210k, BW=103MiB/s '
                                             '(108MB/s)(719MiB/7000msec)\n'
                                             '    slat (nsec): min=1373, '
                                             'max=39404, avg=1621.16, '
                                             'stdev=192.15\n'
                                             '    clat (nsec): min=180, '
                                             'max=110587, avg=7626.16, '
                                             'stdev=2228.40\n'
                                             '     lat (usec): min=5, max=112, '
                                             'avg= 9.25, stdev= 2.26\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 5216],  '
                                             '5.00th=[ 5280], 10.00th=[ 5344], '
                                             '20.00th=[ 5472],\n'
                                             '     | 30.00th=[ 6752], '
                                             '40.00th=[ 7264], 50.00th=[ '
                                             '7456], 60.00th=[ 7712],\n'
                                             '     | 70.00th=[ 8096], '
                                             '80.00th=[ 9280], 90.00th=[ '
                                             '9408], 95.00th=[ 9664],\n'
                                             '     | 99.00th=[16768], '
                                             '99.50th=[18560], '
                                             '99.90th=[23936], '
                                             '99.95th=[24448],\n'
                                             '     | 99.99th=[61696]\n'
                                             '   bw (  KiB/s): min=104853, '
                                             'max=105510, per=100.00%, '
                                             'avg=105237.77, stdev=207.92, '
                                             'samples=13\n'
                                             '   iops        : min=209706, '
                                             'max=211020, avg=210475.77, '
                                             'stdev=415.85, samples=13\n'
                                             '  lat (nsec)   : 250=0.01%\n'
                                             '  lat (usec)   : 4=0.01%, '
                                             '10=97.28%, 20=2.47%, 50=0.22%, '
                                             '100=0.02%\n'
                                             '  lat (usec)   : 250=0.01%\n'
                                             '  cpu          : usr=14.16%, '
                                             'sys=51.89%, ctx=740379, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, '
                                             '2=100.0%, 4=0.0%, 8=0.0%, '
                                             '16=0.0%, 32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=1472941,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=2\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=103MiB/s (108MB/s), '
                                             '103MiB/s-103MiB/s '
                                             '(108MB/s-108MB/s), io=719MiB '
                                             '(754MB), run=7000-7000msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_162.output'),
                                   'state': {'begin': 1666782827.111502,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_2_io_uring-xnvme_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8950073719024658,
                                             'end': 1666782828.0065093,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_162.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_162.state')},
                       'cmd_163': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_512_2_libaio-reference_0.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_163.output'),
                                   'state': {'begin': 1666782829.0462594,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_2_libaio-reference_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8603544235229492,
                                             'end': 1666782829.9066138,
                                             'err': 1,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_163.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_163.state')},
                       'cmd_164': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_164.output'),
                                   'state': {'begin': 1666782829.9089046,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="libaio-reference" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="2" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="libaio" '
                                                    '--filename="/dev/nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_2_libaio-reference_0.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.068029165267944,
                                             'end': 1666782840.9769337,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_164.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_164.state')},
                       'cmd_165': {'output': 'libaio-reference: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=libaio, iodepth=2\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666789931,\n'
                                             '  "timestamp_ms" : '
                                             '1666789931060,\n'
                                             '  "time" : "Wed Oct 26 15:12:11 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"libaio-reference",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"libaio-reference",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "2",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : "libaio",\n'
                                             '        "filename" : '
                                             '"/dev/nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : 799335936,\n'
                                             '        "io_kbytes" : 780601,\n'
                                             '        "bw_bytes" : 114174537,\n'
                                             '        "bw" : 111498,\n'
                                             '        "iops" : 222997.000429,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 1561202,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1442,\n'
                                             '          "max" : 21661,\n'
                                             '          "mean" : 1649.153308,\n'
                                             '          "stddev" : '
                                             '129.768110,\n'
                                             '          "N" : 1561202\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 2645,\n'
                                             '          "max" : 100528,\n'
                                             '          "mean" : 7067.635010,\n'
                                             '          "stddev" : '
                                             '1738.217242,\n'
                                             '          "N" : 1561203,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 5856,\n'
                                             '            "5.000000" : 5920,\n'
                                             '            "10.000000" : 5984,\n'
                                             '            "20.000000" : 6112,\n'
                                             '            "30.000000" : 6880,\n'
                                             '            "40.000000" : 6944,\n'
                                             '            "50.000000" : 7008,\n'
                                             '            "60.000000" : 7136,\n'
                                             '            "70.000000" : 7200,\n'
                                             '            "80.000000" : 7264,\n'
                                             '            "90.000000" : 7392,\n'
                                             '            "95.000000" : 7520,\n'
                                             '            "99.000000" : '
                                             '15936,\n'
                                             '            "99.500000" : '
                                             '18560,\n'
                                             '            "99.900000" : '
                                             '25216,\n'
                                             '            "99.950000" : '
                                             '25728,\n'
                                             '            "99.990000" : 62720\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 6422,\n'
                                             '          "max" : 102221,\n'
                                             '          "mean" : 8716.788250,\n'
                                             '          "stddev" : '
                                             '1739.886131,\n'
                                             '          "N" : 1561203\n'
                                             '        },\n'
                                             '        "bw_min" : 110982,\n'
                                             '        "bw_max" : 112688,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '111528.500000,\n'
                                             '        "bw_dev" : 424.069797,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 221965,\n'
                                             '        "iops_max" : 225376,\n'
                                             '        "iops_mean" : '
                                             '223057.071429,\n'
                                             '        "iops_stddev" : '
                                             '848.040498,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 20.328571,\n'
                                             '      "sys_cpu" : 47.114286,\n'
                                             '      "ctx" : 783978,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 100.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 97.719770,\n'
                                             '        "20" : 1.871763,\n'
                                             '        "50" : 0.379771,\n'
                                             '        "100" : 0.021009,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 2,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ],\n'
                                             '  "disk_util" : [\n'
                                             '    {\n'
                                             '      "name" : "nvme0n1",\n'
                                             '      "read_ios" : 2224647,\n'
                                             '      "write_ios" : 0,\n'
                                             '      "read_merges" : 0,\n'
                                             '      "write_merges" : 0,\n'
                                             '      "read_ticks" : 10878,\n'
                                             '      "write_ticks" : 0,\n'
                                             '      "in_queue" : 10878,\n'
                                             '      "util" : 99.069123\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'libaio-reference: (groupid=0, '
                                             'jobs=1): err= 0: pid=7730: Wed '
                                             'Oct 26 15:12:11 2022\n'
                                             '  read: IOPS=223k, BW=109MiB/s '
                                             '(114MB/s)(762MiB/7001msec)\n'
                                             '    slat (nsec): min=1442, '
                                             'max=21661, avg=1649.15, '
                                             'stdev=129.77\n'
                                             '    clat (usec): min=2, max=100, '
                                             'avg= 7.07, stdev= 1.74\n'
                                             '     lat (usec): min=6, max=102, '
                                             'avg= 8.72, stdev= 1.74\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 5856],  '
                                             '5.00th=[ 5920], 10.00th=[ 5984], '
                                             '20.00th=[ 6112],\n'
                                             '     | 30.00th=[ 6880], '
                                             '40.00th=[ 6944], 50.00th=[ '
                                             '7008], 60.00th=[ 7136],\n'
                                             '     | 70.00th=[ 7200], '
                                             '80.00th=[ 7264], 90.00th=[ '
                                             '7392], 95.00th=[ 7520],\n'
                                             '     | 99.00th=[15936], '
                                             '99.50th=[18560], '
                                             '99.90th=[25216], '
                                             '99.95th=[25728],\n'
                                             '     | 99.99th=[62720]\n'
                                             '   bw (  KiB/s): min=110982, '
                                             'max=112688, per=100.00%, '
                                             'avg=111528.50, stdev=424.07, '
                                             'samples=14\n'
                                             '   iops        : min=221965, '
                                             'max=225376, avg=223057.07, '
                                             'stdev=848.04, samples=14\n'
                                             '  lat (usec)   : 4=0.01%, '
                                             '10=97.72%, 20=1.87%, 50=0.38%, '
                                             '100=0.02%\n'
                                             '  lat (usec)   : 250=0.01%\n'
                                             '  cpu          : usr=20.33%, '
                                             'sys=47.11%, ctx=783978, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, '
                                             '2=100.0%, 4=0.0%, 8=0.0%, '
                                             '16=0.0%, 32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=1561202,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=2\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=109MiB/s (114MB/s), '
                                             '109MiB/s-109MiB/s '
                                             '(114MB/s-114MB/s), io=762MiB '
                                             '(799MB), run=7001-7001msec\n'
                                             '\n'
                                             'Disk stats (read/write):\n'
                                             '  nvme0n1: ios=2224647/0, '
                                             'merge=0/0, ticks=10878/0, '
                                             'in_queue=10878, util=99.07%\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_165.output'),
                                   'state': {'begin': 1666782840.9795122,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_2_libaio-reference_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7958464622497559,
                                             'end': 1666782841.7753587,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_165.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_165.state')},
                       'cmd_166': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_512_2_libaio-reference_1.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_166.output'),
                                   'state': {'begin': 1666782842.787576,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_2_libaio-reference_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8798675537109375,
                                             'end': 1666782843.6674435,
                                             'err': 1,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_166.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_166.state')},
                       'cmd_167': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_167.output'),
                                   'state': {'begin': 1666782843.6697602,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="libaio-reference" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="2" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="libaio" '
                                                    '--filename="/dev/nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_2_libaio-reference_1.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.101510524749756,
                                             'end': 1666782854.7712708,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_167.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_167.state')},
                       'cmd_168': {'output': 'libaio-reference: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=libaio, iodepth=2\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666789944,\n'
                                             '  "timestamp_ms" : '
                                             '1666789944855,\n'
                                             '  "time" : "Wed Oct 26 15:12:24 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"libaio-reference",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"libaio-reference",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "2",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : "libaio",\n'
                                             '        "filename" : '
                                             '"/dev/nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : 815441920,\n'
                                             '        "io_kbytes" : 796330,\n'
                                             '        "bw_bytes" : 116475063,\n'
                                             '        "bw" : 113745,\n'
                                             '        "iops" : 227490.072847,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 1592658,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1472,\n'
                                             '          "max" : 20098,\n'
                                             '          "mean" : 1679.534212,\n'
                                             '          "stddev" : '
                                             '135.170532,\n'
                                             '          "N" : 1592659\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 431,\n'
                                             '          "max" : 91642,\n'
                                             '          "mean" : 6852.600837,\n'
                                             '          "stddev" : '
                                             '1799.750306,\n'
                                             '          "N" : 1592659,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 5856,\n'
                                             '            "5.000000" : 5920,\n'
                                             '            "10.000000" : 5920,\n'
                                             '            "20.000000" : 6048,\n'
                                             '            "30.000000" : 6112,\n'
                                             '            "40.000000" : 6240,\n'
                                             '            "50.000000" : 6368,\n'
                                             '            "60.000000" : 7008,\n'
                                             '            "70.000000" : 7136,\n'
                                             '            "80.000000" : 7328,\n'
                                             '            "90.000000" : 7456,\n'
                                             '            "95.000000" : 7584,\n'
                                             '            "99.000000" : '
                                             '15680,\n'
                                             '            "99.500000" : '
                                             '18560,\n'
                                             '            "99.900000" : '
                                             '25472,\n'
                                             '            "99.950000" : '
                                             '25984,\n'
                                             '            "99.990000" : 62720\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 5681,\n'
                                             '          "max" : 93386,\n'
                                             '          "mean" : 8532.135048,\n'
                                             '          "stddev" : '
                                             '1803.334792,\n'
                                             '          "N" : 1592659\n'
                                             '        },\n'
                                             '        "bw_min" : 113374,\n'
                                             '        "bw_max" : 114111,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '113776.000000,\n'
                                             '        "bw_dev" : 230.273082,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 226748,\n'
                                             '        "iops_max" : 228222,\n'
                                             '        "iops_mean" : '
                                             '227551.928571,\n'
                                             '        "iops_stddev" : '
                                             '460.425633,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 23.000000,\n'
                                             '      "sys_cpu" : 47.142857,\n'
                                             '      "ctx" : 800025,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 1,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 100.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.010000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 97.734856,\n'
                                             '        "20" : 1.887411,\n'
                                             '        "50" : 0.346276,\n'
                                             '        "100" : 0.023420,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 2,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ],\n'
                                             '  "disk_util" : [\n'
                                             '    {\n'
                                             '      "name" : "nvme0n1",\n'
                                             '      "read_ios" : 2265813,\n'
                                             '      "write_ios" : 0,\n'
                                             '      "read_merges" : 0,\n'
                                             '      "write_merges" : 0,\n'
                                             '      "read_ticks" : 11215,\n'
                                             '      "write_ticks" : 0,\n'
                                             '      "in_queue" : 11215,\n'
                                             '      "util" : 99.069123\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'libaio-reference: (groupid=0, '
                                             'jobs=1): err= 0: pid=7778: Wed '
                                             'Oct 26 15:12:24 2022\n'
                                             '  read: IOPS=227k, BW=111MiB/s '
                                             '(116MB/s)(778MiB/7001msec)\n'
                                             '    slat (nsec): min=1472, '
                                             'max=20098, avg=1679.53, '
                                             'stdev=135.17\n'
                                             '    clat (nsec): min=431, '
                                             'max=91642, avg=6852.60, '
                                             'stdev=1799.75\n'
                                             '     lat (nsec): min=5681, '
                                             'max=93386, avg=8532.14, '
                                             'stdev=1803.33\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 5856],  '
                                             '5.00th=[ 5920], 10.00th=[ 5920], '
                                             '20.00th=[ 6048],\n'
                                             '     | 30.00th=[ 6112], '
                                             '40.00th=[ 6240], 50.00th=[ '
                                             '6368], 60.00th=[ 7008],\n'
                                             '     | 70.00th=[ 7136], '
                                             '80.00th=[ 7328], 90.00th=[ '
                                             '7456], 95.00th=[ 7584],\n'
                                             '     | 99.00th=[15680], '
                                             '99.50th=[18560], '
                                             '99.90th=[25472], '
                                             '99.95th=[25984],\n'
                                             '     | 99.99th=[62720]\n'
                                             '   bw (  KiB/s): min=113374, '
                                             'max=114111, per=100.00%, '
                                             'avg=113776.00, stdev=230.27, '
                                             'samples=14\n'
                                             '   iops        : min=226748, '
                                             'max=228222, avg=227551.93, '
                                             'stdev=460.43, samples=14\n'
                                             '  lat (nsec)   : 500=0.01%\n'
                                             '  lat (usec)   : 4=0.01%, '
                                             '10=97.73%, 20=1.89%, 50=0.35%, '
                                             '100=0.02%\n'
                                             '  cpu          : usr=23.00%, '
                                             'sys=47.14%, ctx=800025, majf=0, '
                                             'minf=1\n'
                                             '  IO depths    : 1=0.0%, '
                                             '2=100.0%, 4=0.0%, 8=0.0%, '
                                             '16=0.0%, 32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=1592658,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=2\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=111MiB/s (116MB/s), '
                                             '111MiB/s-111MiB/s '
                                             '(116MB/s-116MB/s), io=778MiB '
                                             '(815MB), run=7001-7001msec\n'
                                             '\n'
                                             'Disk stats (read/write):\n'
                                             '  nvme0n1: ios=2265813/0, '
                                             'merge=0/0, ticks=11215/0, '
                                             'in_queue=11215, util=99.07%\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_168.output'),
                                   'state': {'begin': 1666782854.7738547,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_2_libaio-reference_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8583071231842041,
                                             'end': 1666782855.6321619,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_168.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_168.state')},
                       'cmd_169': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_512_2_libaio-reference_2.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_169.output'),
                                   'state': {'begin': 1666782856.687171,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_2_libaio-reference_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8172504901885986,
                                             'end': 1666782857.5044215,
                                             'err': 1,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_169.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_169.state')},
                       'cmd_17': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_17.output'),
                                  'state': {'begin': 1666782116.380387,
                                            'cmd': '/opt/fio/bin/fio '
                                                   '--name="io_uring_cmd-xnvme" '
                                                   '--rw="randread" '
                                                   '--size="12G" --iodepth="1" '
                                                   '--bs="512" --direct="0" '
                                                   '--thread="1" '
                                                   '--time_based="1" '
                                                   '--runtime="7" '
                                                   '--ramp_time="3" '
                                                   '--norandommap="1" '
                                                   '--allow_file_create="0" '
                                                   '--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
                                                   '--xnvme_be="linux" '
                                                   '--xnvme_async="io_uring_cmd" '
                                                   '--xnvme_sync="nvme" '
                                                   '--xnvme_admin="nvme" '
                                                   '--xnvme_dev_nsid="1" '
                                                   '--filename="/dev/ng0n1" '
                                                   '--output-format="normal,json" '
                                                   '--output="/tmp/fio-output_512_1_io_uring_cmd-xnvme_2.txt" '
                                                   '--cpus_allowed="1"',
                                            'cwd': 'None',
                                            'elapsed': 11.070308208465576,
                                            'end': 1666782127.4506953,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_17.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_17.state')},
                       'cmd_170': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_170.output'),
                                   'state': {'begin': 1666782857.506634,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="libaio-reference" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="2" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="libaio" '
                                                    '--filename="/dev/nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_2_libaio-reference_2.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.087865829467773,
                                             'end': 1666782868.5944998,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_170.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_170.state')},
                       'cmd_171': {'output': 'libaio-reference: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=libaio, iodepth=2\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666789958,\n'
                                             '  "timestamp_ms" : '
                                             '1666789958677,\n'
                                             '  "time" : "Wed Oct 26 15:12:38 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"libaio-reference",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"libaio-reference",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "2",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : "libaio",\n'
                                             '        "filename" : '
                                             '"/dev/nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : 812047872,\n'
                                             '        "io_kbytes" : 793015,\n'
                                             '        "bw_bytes" : 115990268,\n'
                                             '        "bw" : 113271,\n'
                                             '        "iops" : 226543.208113,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 1586029,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1452,\n'
                                             '          "max" : 14798,\n'
                                             '          "mean" : 1655.304542,\n'
                                             '          "stddev" : '
                                             '123.996712,\n'
                                             '          "N" : 1586030\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 421,\n'
                                             '          "max" : 120085,\n'
                                             '          "mean" : 6916.411781,\n'
                                             '          "stddev" : '
                                             '1733.488478,\n'
                                             '          "N" : 1586030,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 5792,\n'
                                             '            "5.000000" : 5856,\n'
                                             '            "10.000000" : 5920,\n'
                                             '            "20.000000" : 6048,\n'
                                             '            "30.000000" : 6112,\n'
                                             '            "40.000000" : 6816,\n'
                                             '            "50.000000" : 6944,\n'
                                             '            "60.000000" : 7008,\n'
                                             '            "70.000000" : 7136,\n'
                                             '            "80.000000" : 7264,\n'
                                             '            "90.000000" : 7328,\n'
                                             '            "95.000000" : 7520,\n'
                                             '            "99.000000" : '
                                             '15680,\n'
                                             '            "99.500000" : '
                                             '18560,\n'
                                             '            "99.900000" : '
                                             '24192,\n'
                                             '            "99.950000" : '
                                             '25728,\n'
                                             '            "99.990000" : 61184\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 5661,\n'
                                             '          "max" : 121798,\n'
                                             '          "mean" : 8571.716323,\n'
                                             '          "stddev" : '
                                             '1737.367517,\n'
                                             '          "N" : 1586030\n'
                                             '        },\n'
                                             '        "bw_min" : 112211,\n'
                                             '        "bw_max" : 114174,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '113343.571429,\n'
                                             '        "bw_dev" : 546.633292,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 224422,\n'
                                             '        "iops_max" : 228348,\n'
                                             '        "iops_mean" : '
                                             '226687.285714,\n'
                                             '        "iops_stddev" : '
                                             '1093.360904,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 21.314286,\n'
                                             '      "sys_cpu" : 47.257143,\n'
                                             '      "ctx" : 796376,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 100.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.010000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 97.729802,\n'
                                             '        "20" : 1.891264,\n'
                                             '        "50" : 0.351885,\n'
                                             '        "100" : 0.018789,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 2,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ],\n'
                                             '  "disk_util" : [\n'
                                             '    {\n'
                                             '      "name" : "nvme0n1",\n'
                                             '      "read_ios" : 2250040,\n'
                                             '      "write_ios" : 0,\n'
                                             '      "read_merges" : 0,\n'
                                             '      "write_merges" : 0,\n'
                                             '      "read_ticks" : 11049,\n'
                                             '      "write_ticks" : 0,\n'
                                             '      "in_queue" : 11049,\n'
                                             '      "util" : 99.108381\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'libaio-reference: (groupid=0, '
                                             'jobs=1): err= 0: pid=7824: Wed '
                                             'Oct 26 15:12:38 2022\n'
                                             '  read: IOPS=227k, BW=111MiB/s '
                                             '(116MB/s)(774MiB/7001msec)\n'
                                             '    slat (nsec): min=1452, '
                                             'max=14798, avg=1655.30, '
                                             'stdev=124.00\n'
                                             '    clat (nsec): min=421, '
                                             'max=120085, avg=6916.41, '
                                             'stdev=1733.49\n'
                                             '     lat (usec): min=5, max=121, '
                                             'avg= 8.57, stdev= 1.74\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 5792],  '
                                             '5.00th=[ 5856], 10.00th=[ 5920], '
                                             '20.00th=[ 6048],\n'
                                             '     | 30.00th=[ 6112], '
                                             '40.00th=[ 6816], 50.00th=[ '
                                             '6944], 60.00th=[ 7008],\n'
                                             '     | 70.00th=[ 7136], '
                                             '80.00th=[ 7264], 90.00th=[ '
                                             '7328], 95.00th=[ 7520],\n'
                                             '     | 99.00th=[15680], '
                                             '99.50th=[18560], '
                                             '99.90th=[24192], '
                                             '99.95th=[25728],\n'
                                             '     | 99.99th=[61184]\n'
                                             '   bw (  KiB/s): min=112211, '
                                             'max=114174, per=100.00%, '
                                             'avg=113343.57, stdev=546.63, '
                                             'samples=14\n'
                                             '   iops        : min=224422, '
                                             'max=228348, avg=226687.29, '
                                             'stdev=1093.36, samples=14\n'
                                             '  lat (nsec)   : 500=0.01%\n'
                                             '  lat (usec)   : 4=0.01%, '
                                             '10=97.73%, 20=1.89%, 50=0.35%, '
                                             '100=0.02%\n'
                                             '  lat (usec)   : 250=0.01%\n'
                                             '  cpu          : usr=21.31%, '
                                             'sys=47.26%, ctx=796376, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, '
                                             '2=100.0%, 4=0.0%, 8=0.0%, '
                                             '16=0.0%, 32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=1586029,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=2\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=111MiB/s (116MB/s), '
                                             '111MiB/s-111MiB/s '
                                             '(116MB/s-116MB/s), io=774MiB '
                                             '(812MB), run=7001-7001msec\n'
                                             '\n'
                                             'Disk stats (read/write):\n'
                                             '  nvme0n1: ios=2250040/0, '
                                             'merge=0/0, ticks=11049/0, '
                                             'in_queue=11049, util=99.11%\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_171.output'),
                                   'state': {'begin': 1666782868.597043,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_2_libaio-reference_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8535284996032715,
                                             'end': 1666782869.4505715,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_171.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_171.state')},
                       'cmd_172': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_512_2_libaio-bdev_aio_0.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_172.output'),
                                   'state': {'begin': 1666782871.6327507,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_2_libaio-bdev_aio_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.939302921295166,
                                             'end': 1666782872.5720537,
                                             'err': 1,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_172.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_172.state')},
                       'cmd_173': {'output': 'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_173.output'),
                                   'state': {'begin': 1666782872.5743666,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="libaio-bdev_aio" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="2" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="spdk_bdev" '
                                                    '--spdk_conf="/tmp/spdk_bdev_aio_foo.conf" '
                                                    '--filename="Nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_2_libaio-bdev_aio_0.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.872151613235474,
                                             'end': 1666782884.4465182,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_173.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_173.state')},
                       'cmd_174': {'output': 'libaio-bdev_aio: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=spdk_bdev, iodepth=2\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666789974,\n'
                                             '  "timestamp_ms" : '
                                             '1666789974420,\n'
                                             '  "time" : "Wed Oct 26 15:12:54 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"libaio-bdev_aio",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"libaio-bdev_aio",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "2",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"spdk_bdev",\n'
                                             '        "spdk_conf" : '
                                             '"/tmp/spdk_bdev_aio_foo.conf",\n'
                                             '        "filename" : "Nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '1065744896,\n'
                                             '        "io_kbytes" : 1040766,\n'
                                             '        "bw_bytes" : 152227524,\n'
                                             '        "bw" : 148659,\n'
                                             '        "iops" : 297319.240109,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 2081532,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1663,\n'
                                             '          "max" : 22833,\n'
                                             '          "mean" : 1889.934944,\n'
                                             '          "stddev" : '
                                             '127.676401,\n'
                                             '          "N" : 2081532\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 241,\n'
                                             '          "max" : 102011,\n'
                                             '          "mean" : 4605.353980,\n'
                                             '          "stddev" : '
                                             '1741.728440,\n'
                                             '          "N" : 2081533,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 3952,\n'
                                             '            "5.000000" : 4192,\n'
                                             '            "10.000000" : 4256,\n'
                                             '            "20.000000" : 4256,\n'
                                             '            "30.000000" : 4320,\n'
                                             '            "40.000000" : 4320,\n'
                                             '            "50.000000" : 4384,\n'
                                             '            "60.000000" : 4384,\n'
                                             '            "70.000000" : 4384,\n'
                                             '            "80.000000" : 4448,\n'
                                             '            "90.000000" : 4512,\n'
                                             '            "95.000000" : 4704,\n'
                                             '            "99.000000" : '
                                             '12608,\n'
                                             '            "99.500000" : '
                                             '17536,\n'
                                             '            "99.900000" : '
                                             '21888,\n'
                                             '            "99.950000" : '
                                             '22400,\n'
                                             '            "99.990000" : 59648\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 5129,\n'
                                             '          "max" : 103885,\n'
                                             '          "mean" : 6495.288902,\n'
                                             '          "stddev" : '
                                             '1745.563229,\n'
                                             '          "N" : 2081533\n'
                                             '        },\n'
                                             '        "bw_min" : 147611,\n'
                                             '        "bw_max" : 149598,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '148687.142857,\n'
                                             '        "bw_dev" : 617.469882,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 295222,\n'
                                             '        "iops_max" : 299196,\n'
                                             '        "iops_mean" : '
                                             '297374.357143,\n'
                                             '        "iops_stddev" : '
                                             '1234.727102,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 63.985714,\n'
                                             '      "sys_cpu" : 35.985714,\n'
                                             '      "ctx" : 15,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.100000,\n'
                                             '        "2" : 99.989959,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.010000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 2.077076,\n'
                                             '        "10" : 95.930641,\n'
                                             '        "20" : 1.729688,\n'
                                             '        "50" : 0.242177,\n'
                                             '        "100" : 0.017968,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 2,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'libaio-bdev_aio: (groupid=0, '
                                             'jobs=1): err= 0: pid=7881: Wed '
                                             'Oct 26 15:12:54 2022\n'
                                             '  read: IOPS=297k, BW=145MiB/s '
                                             '(152MB/s)(1016MiB/7001msec)\n'
                                             '    slat (nsec): min=1663, '
                                             'max=22833, avg=1889.93, '
                                             'stdev=127.68\n'
                                             '    clat (nsec): min=241, '
                                             'max=102011, avg=4605.35, '
                                             'stdev=1741.73\n'
                                             '     lat (usec): min=5, max=103, '
                                             'avg= 6.50, stdev= 1.75\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 3952],  '
                                             '5.00th=[ 4192], 10.00th=[ 4256], '
                                             '20.00th=[ 4256],\n'
                                             '     | 30.00th=[ 4320], '
                                             '40.00th=[ 4320], 50.00th=[ '
                                             '4384], 60.00th=[ 4384],\n'
                                             '     | 70.00th=[ 4384], '
                                             '80.00th=[ 4448], 90.00th=[ '
                                             '4512], 95.00th=[ 4704],\n'
                                             '     | 99.00th=[12608], '
                                             '99.50th=[17536], '
                                             '99.90th=[21888], '
                                             '99.95th=[22400],\n'
                                             '     | 99.99th=[59648]\n'
                                             '   bw (  KiB/s): min=147611, '
                                             'max=149598, per=100.00%, '
                                             'avg=148687.14, stdev=617.47, '
                                             'samples=14\n'
                                             '   iops        : min=295222, '
                                             'max=299196, avg=297374.36, '
                                             'stdev=1234.73, samples=14\n'
                                             '  lat (nsec)   : 250=0.01%, '
                                             '500=0.01%\n'
                                             '  lat (usec)   : 4=2.08%, '
                                             '10=95.93%, 20=1.73%, 50=0.24%, '
                                             '100=0.02%\n'
                                             '  lat (usec)   : 250=0.01%\n'
                                             '  cpu          : usr=63.99%, '
                                             'sys=35.99%, ctx=15, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.1%, '
                                             '2=100.0%, 4=0.0%, 8=0.0%, '
                                             '16=0.0%, 32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=2081532,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=2\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=145MiB/s (152MB/s), '
                                             '145MiB/s-145MiB/s '
                                             '(152MB/s-152MB/s), io=1016MiB '
                                             '(1066MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_174.output'),
                                   'state': {'begin': 1666782884.4493692,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_2_libaio-bdev_aio_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8834080696105957,
                                             'end': 1666782885.3327773,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_174.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_174.state')},
                       'cmd_175': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_512_2_libaio-bdev_aio_1.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_175.output'),
                                   'state': {'begin': 1666782887.4688983,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_2_libaio-bdev_aio_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8365371227264404,
                                             'end': 1666782888.3054354,
                                             'err': 1,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_175.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_175.state')},
                       'cmd_176': {'output': 'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_176.output'),
                                   'state': {'begin': 1666782888.307591,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="libaio-bdev_aio" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="2" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="spdk_bdev" '
                                                    '--spdk_conf="/tmp/spdk_bdev_aio_foo.conf" '
                                                    '--filename="Nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_2_libaio-bdev_aio_1.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.422178983688354,
                                             'end': 1666782899.72977,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_176.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_176.state')},
                       'cmd_177': {'output': 'libaio-bdev_aio: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=spdk_bdev, iodepth=2\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666789989,\n'
                                             '  "timestamp_ms" : '
                                             '1666789989718,\n'
                                             '  "time" : "Wed Oct 26 15:13:09 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"libaio-bdev_aio",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"libaio-bdev_aio",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "2",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"spdk_bdev",\n'
                                             '        "spdk_conf" : '
                                             '"/tmp/spdk_bdev_aio_foo.conf",\n'
                                             '        "filename" : "Nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '1061161472,\n'
                                             '        "io_kbytes" : 1036290,\n'
                                             '        "bw_bytes" : 151572842,\n'
                                             '        "bw" : 148020,\n'
                                             '        "iops" : 296040.422797,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 2072579,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1683,\n'
                                             '          "max" : 35486,\n'
                                             '          "mean" : 1927.547332,\n'
                                             '          "stddev" : '
                                             '125.332926,\n'
                                             '          "N" : 2072580\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 240,\n'
                                             '          "max" : 98986,\n'
                                             '          "mean" : 4604.571840,\n'
                                             '          "stddev" : '
                                             '1756.957642,\n'
                                             '          "N" : 2072579,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 3952,\n'
                                             '            "5.000000" : 4192,\n'
                                             '            "10.000000" : 4256,\n'
                                             '            "20.000000" : 4256,\n'
                                             '            "30.000000" : 4320,\n'
                                             '            "40.000000" : 4320,\n'
                                             '            "50.000000" : 4384,\n'
                                             '            "60.000000" : 4384,\n'
                                             '            "70.000000" : 4384,\n'
                                             '            "80.000000" : 4448,\n'
                                             '            "90.000000" : 4512,\n'
                                             '            "95.000000" : 4704,\n'
                                             '            "99.000000" : '
                                             '12608,\n'
                                             '            "99.500000" : '
                                             '17536,\n'
                                             '            "99.900000" : '
                                             '21888,\n'
                                             '            "99.950000" : '
                                             '22400,\n'
                                             '            "99.990000" : 61184\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 5180,\n'
                                             '          "max" : 100919,\n'
                                             '          "mean" : 6532.119198,\n'
                                             '          "stddev" : '
                                             '1759.977965,\n'
                                             '          "N" : 2072579\n'
                                             '        },\n'
                                             '        "bw_min" : 147349,\n'
                                             '        "bw_max" : 148689,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '148046.357143,\n'
                                             '        "bw_dev" : 438.091070,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 294698,\n'
                                             '        "iops_max" : 297378,\n'
                                             '        "iops_mean" : '
                                             '296092.714286,\n'
                                             '        "iops_stddev" : '
                                             '876.182140,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 63.528571,\n'
                                             '      "sys_cpu" : 36.442857,\n'
                                             '      "ctx" : 11,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.100000,\n'
                                             '        "2" : 99.989240,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.010000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 2.120884,\n'
                                             '        "10" : 95.873450,\n'
                                             '        "20" : 1.738317,\n'
                                             '        "50" : 0.244237,\n'
                                             '        "100" : 0.020506,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 2,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'libaio-bdev_aio: (groupid=0, '
                                             'jobs=1): err= 0: pid=7940: Wed '
                                             'Oct 26 15:13:09 2022\n'
                                             '  read: IOPS=296k, BW=145MiB/s '
                                             '(152MB/s)(1012MiB/7001msec)\n'
                                             '    slat (nsec): min=1683, '
                                             'max=35486, avg=1927.55, '
                                             'stdev=125.33\n'
                                             '    clat (nsec): min=240, '
                                             'max=98986, avg=4604.57, '
                                             'stdev=1756.96\n'
                                             '     lat (usec): min=5, max=100, '
                                             'avg= 6.53, stdev= 1.76\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 3952],  '
                                             '5.00th=[ 4192], 10.00th=[ 4256], '
                                             '20.00th=[ 4256],\n'
                                             '     | 30.00th=[ 4320], '
                                             '40.00th=[ 4320], 50.00th=[ '
                                             '4384], 60.00th=[ 4384],\n'
                                             '     | 70.00th=[ 4384], '
                                             '80.00th=[ 4448], 90.00th=[ '
                                             '4512], 95.00th=[ 4704],\n'
                                             '     | 99.00th=[12608], '
                                             '99.50th=[17536], '
                                             '99.90th=[21888], '
                                             '99.95th=[22400],\n'
                                             '     | 99.99th=[61184]\n'
                                             '   bw (  KiB/s): min=147349, '
                                             'max=148689, per=100.00%, '
                                             'avg=148046.36, stdev=438.09, '
                                             'samples=14\n'
                                             '   iops        : min=294698, '
                                             'max=297378, avg=296092.71, '
                                             'stdev=876.18, samples=14\n'
                                             '  lat (nsec)   : 250=0.01%, '
                                             '500=0.01%\n'
                                             '  lat (usec)   : 4=2.12%, '
                                             '10=95.87%, 20=1.74%, 50=0.24%, '
                                             '100=0.02%\n'
                                             '  cpu          : usr=63.53%, '
                                             'sys=36.44%, ctx=11, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.1%, '
                                             '2=100.0%, 4=0.0%, 8=0.0%, '
                                             '16=0.0%, 32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=2072579,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=2\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=145MiB/s (152MB/s), '
                                             '145MiB/s-145MiB/s '
                                             '(152MB/s-152MB/s), io=1012MiB '
                                             '(1061MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_177.output'),
                                   'state': {'begin': 1666782899.732175,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_2_libaio-bdev_aio_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8234589099884033,
                                             'end': 1666782900.555634,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_177.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_177.state')},
                       'cmd_178': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_512_2_libaio-bdev_aio_2.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_178.output'),
                                   'state': {'begin': 1666782902.744454,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_2_libaio-bdev_aio_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8046927452087402,
                                             'end': 1666782903.5491467,
                                             'err': 1,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_178.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_178.state')},
                       'cmd_179': {'output': 'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_179.output'),
                                   'state': {'begin': 1666782903.551061,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="libaio-bdev_aio" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="2" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="spdk_bdev" '
                                                    '--spdk_conf="/tmp/spdk_bdev_aio_foo.conf" '
                                                    '--filename="Nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_2_libaio-bdev_aio_2.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.563740968704224,
                                             'end': 1666782915.114802,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_179.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_179.state')},
                       'cmd_18': {'output': 'io_uring_cmd-xnvme: (g=0): '
                                            'rw=randread, bs=(R) 512B-512B, '
                                            '(W) 512B-512B, (T) 512B-512B, '
                                            'ioengine=xnvme, iodepth=1\n'
                                            'fio-3.32-dirty\n'
                                            'Starting 1 thread\n'
                                            '{\n'
                                            '  "fio version" : '
                                            '"fio-3.32-dirty",\n'
                                            '  "timestamp" : 1666789217,\n'
                                            '  "timestamp_ms" : '
                                            '1666789217545,\n'
                                            '  "time" : "Wed Oct 26 15:00:17 '
                                            '2022",\n'
                                            '  "jobs" : [\n'
                                            '    {\n'
                                            '      "jobname" : '
                                            '"io_uring_cmd-xnvme",\n'
                                            '      "groupid" : 0,\n'
                                            '      "error" : 0,\n'
                                            '      "eta" : 0,\n'
                                            '      "elapsed" : 11,\n'
                                            '      "job options" : {\n'
                                            '        "name" : '
                                            '"io_uring_cmd-xnvme",\n'
                                            '        "rw" : "randread",\n'
                                            '        "size" : "12G",\n'
                                            '        "iodepth" : "1",\n'
                                            '        "bs" : "512",\n'
                                            '        "direct" : "0",\n'
                                            '        "thread" : "1",\n'
                                            '        "time_based" : "1",\n'
                                            '        "runtime" : "7",\n'
                                            '        "ramp_time" : "3",\n'
                                            '        "norandommap" : "1",\n'
                                            '        "allow_file_create" : '
                                            '"0",\n'
                                            '        "ioengine" : '
                                            '"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
                                            '        "xnvme_be" : "linux",\n'
                                            '        "xnvme_async" : '
                                            '"io_uring_cmd",\n'
                                            '        "xnvme_sync" : "nvme",\n'
                                            '        "xnvme_admin" : "nvme",\n'
                                            '        "xnvme_dev_nsid" : "1",\n'
                                            '        "filename" : '
                                            '"/dev/ng0n1",\n'
                                            '        "cpus_allowed" : "1"\n'
                                            '      },\n'
                                            '      "read" : {\n'
                                            '        "io_bytes" : 392771072,\n'
                                            '        "io_kbytes" : 383565,\n'
                                            '        "bw_bytes" : 56102138,\n'
                                            '        "bw" : 54787,\n'
                                            '        "iops" : 109574.489359,\n'
                                            '        "runtime" : 7001,\n'
                                            '        "total_ios" : 767131,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 1423,\n'
                                            '          "max" : 5861,\n'
                                            '          "mean" : 1520.401970,\n'
                                            '          "stddev" : 57.282899,\n'
                                            '          "N" : 767131\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 1362,\n'
                                            '          "max" : 75181,\n'
                                            '          "mean" : 7308.434213,\n'
                                            '          "stddev" : 994.829189,\n'
                                            '          "N" : 767131,\n'
                                            '          "percentile" : {\n'
                                            '            "1.000000" : 6816,\n'
                                            '            "5.000000" : 7008,\n'
                                            '            "10.000000" : 7072,\n'
                                            '            "20.000000" : 7072,\n'
                                            '            "30.000000" : 7136,\n'
                                            '            "40.000000" : 7200,\n'
                                            '            "50.000000" : 7200,\n'
                                            '            "60.000000" : 7264,\n'
                                            '            "70.000000" : 7264,\n'
                                            '            "80.000000" : 7328,\n'
                                            '            "90.000000" : 7456,\n'
                                            '            "95.000000" : 7520,\n'
                                            '            "99.000000" : 10432,\n'
                                            '            "99.500000" : 13632,\n'
                                            '            "99.900000" : 15680,\n'
                                            '            "99.950000" : 15808,\n'
                                            '            "99.990000" : 59136\n'
                                            '          }\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 6832,\n'
                                            '          "max" : 76674,\n'
                                            '          "mean" : 8828.836183,\n'
                                            '          "stddev" : 997.829487,\n'
                                            '          "N" : 767131\n'
                                            '        },\n'
                                            '        "bw_min" : 54620,\n'
                                            '        "bw_max" : 55051,\n'
                                            '        "bw_agg" : 100.000000,\n'
                                            '        "bw_mean" : '
                                            '54797.071429,\n'
                                            '        "bw_dev" : 124.224031,\n'
                                            '        "bw_samples" : 14,\n'
                                            '        "iops_min" : 109240,\n'
                                            '        "iops_max" : 110102,\n'
                                            '        "iops_mean" : '
                                            '109594.142857,\n'
                                            '        "iops_stddev" : '
                                            '248.448062,\n'
                                            '        "iops_samples" : 14\n'
                                            '      },\n'
                                            '      "write" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "trim" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "sync" : {\n'
                                            '        "total_ios" : 0,\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        }\n'
                                            '      },\n'
                                            '      "job_runtime" : 7000,\n'
                                            '      "usr_cpu" : 7.342857,\n'
                                            '      "sys_cpu" : 37.014286,\n'
                                            '      "ctx" : 767130,\n'
                                            '      "majf" : 0,\n'
                                            '      "minf" : 0,\n'
                                            '      "iodepth_level" : {\n'
                                            '        "1" : 100.000000,\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_submit" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_complete" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ns" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_us" : {\n'
                                            '        "2" : 0.010000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 98.706740,\n'
                                            '        "20" : 1.275532,\n'
                                            '        "50" : 0.010000,\n'
                                            '        "100" : 0.014861,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ms" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000,\n'
                                            '        "2000" : 0.000000,\n'
                                            '        ">=2000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_depth" : 1,\n'
                                            '      "latency_target" : 0,\n'
                                            '      "latency_percentile" : '
                                            '100.000000,\n'
                                            '      "latency_window" : 0\n'
                                            '    }\n'
                                            '  ]\n'
                                            '}\n'
                                            '\n'
                                            'io_uring_cmd-xnvme: (groupid=0, '
                                            'jobs=1): err= 0: pid=5169: Wed '
                                            'Oct 26 15:00:17 2022\n'
                                            '  read: IOPS=110k, BW=53.5MiB/s '
                                            '(56.1MB/s)(375MiB/7001msec)\n'
                                            '    slat (nsec): min=1423, '
                                            'max=5861, avg=1520.40, '
                                            'stdev=57.28\n'
                                            '    clat (nsec): min=1362, '
                                            'max=75181, avg=7308.43, '
                                            'stdev=994.83\n'
                                            '     lat (nsec): min=6832, '
                                            'max=76674, avg=8828.84, '
                                            'stdev=997.83\n'
                                            '    clat percentiles (nsec):\n'
                                            '     |  1.00th=[ 6816],  5.00th=[ '
                                            '7008], 10.00th=[ 7072], 20.00th=[ '
                                            '7072],\n'
                                            '     | 30.00th=[ 7136], 40.00th=[ '
                                            '7200], 50.00th=[ 7200], 60.00th=[ '
                                            '7264],\n'
                                            '     | 70.00th=[ 7264], 80.00th=[ '
                                            '7328], 90.00th=[ 7456], 95.00th=[ '
                                            '7520],\n'
                                            '     | 99.00th=[10432], '
                                            '99.50th=[13632], 99.90th=[15680], '
                                            '99.95th=[15808],\n'
                                            '     | 99.99th=[59136]\n'
                                            '   bw (  KiB/s): min=54620, '
                                            'max=55051, per=100.00%, '
                                            'avg=54797.07, stdev=124.22, '
                                            'samples=14\n'
                                            '   iops        : min=109240, '
                                            'max=110102, avg=109594.14, '
                                            'stdev=248.45, samples=14\n'
                                            '  lat (usec)   : 2=0.01%, '
                                            '10=98.71%, 20=1.28%, 50=0.01%, '
                                            '100=0.01%\n'
                                            '  cpu          : usr=7.34%, '
                                            'sys=37.01%, ctx=767130, majf=0, '
                                            'minf=0\n'
                                            '  IO depths    : 1=100.0%, '
                                            '2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, >=64=0.0%\n'
                                            '     submit    : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     complete  : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     issued rwts: '
                                            'total=767131,0,0,0 short=0,0,0,0 '
                                            'dropped=0,0,0,0\n'
                                            '     latency   : target=0, '
                                            'window=0, percentile=100.00%, '
                                            'depth=1\n'
                                            '\n'
                                            'Run status group 0 (all jobs):\n'
                                            '   READ: bw=53.5MiB/s (56.1MB/s), '
                                            '53.5MiB/s-53.5MiB/s '
                                            '(56.1MB/s-56.1MB/s), io=375MiB '
                                            '(393MB), run=7001-7001msec\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_18.output'),
                                  'state': {'begin': 1666782127.4535794,
                                            'cmd': 'cat '
                                                   '/tmp/fio-output_512_1_io_uring_cmd-xnvme_2.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.8653717041015625,
                                            'end': 1666782128.3189511,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_18.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_18.state')},
                       'cmd_180': {'output': 'libaio-bdev_aio: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=spdk_bdev, iodepth=2\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666790005,\n'
                                             '  "timestamp_ms" : '
                                             '1666790005075,\n'
                                             '  "time" : "Wed Oct 26 15:13:25 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"libaio-bdev_aio",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"libaio-bdev_aio",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "2",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"spdk_bdev",\n'
                                             '        "spdk_conf" : '
                                             '"/tmp/spdk_bdev_aio_foo.conf",\n'
                                             '        "filename" : "Nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '1064973824,\n'
                                             '        "io_kbytes" : 1040013,\n'
                                             '        "bw_bytes" : 152117386,\n'
                                             '        "bw" : 148552,\n'
                                             '        "iops" : 297103.985145,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 2080025,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1663,\n'
                                             '          "max" : 18003,\n'
                                             '          "mean" : 1897.384686,\n'
                                             '          "stddev" : '
                                             '130.276176,\n'
                                             '          "N" : 2080026\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 250,\n'
                                             '          "max" : 117160,\n'
                                             '          "mean" : 4608.007251,\n'
                                             '          "stddev" : '
                                             '1767.883717,\n'
                                             '          "N" : 2080025,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 3952,\n'
                                             '            "5.000000" : 4192,\n'
                                             '            "10.000000" : 4256,\n'
                                             '            "20.000000" : 4256,\n'
                                             '            "30.000000" : 4320,\n'
                                             '            "40.000000" : 4320,\n'
                                             '            "50.000000" : 4384,\n'
                                             '            "60.000000" : 4384,\n'
                                             '            "70.000000" : 4448,\n'
                                             '            "80.000000" : 4448,\n'
                                             '            "90.000000" : 4512,\n'
                                             '            "95.000000" : 4704,\n'
                                             '            "99.000000" : '
                                             '12608,\n'
                                             '            "99.500000" : '
                                             '17536,\n'
                                             '            "99.900000" : '
                                             '21888,\n'
                                             '            "99.950000" : '
                                             '22400,\n'
                                             '            "99.990000" : 61184\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 5160,\n'
                                             '          "max" : 119033,\n'
                                             '          "mean" : 6505.391929,\n'
                                             '          "stddev" : '
                                             '1771.826567,\n'
                                             '          "N" : 2080025\n'
                                             '        },\n'
                                             '        "bw_min" : 147892,\n'
                                             '        "bw_max" : 149767,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '148627.928571,\n'
                                             '        "bw_dev" : 466.567740,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 295785,\n'
                                             '        "iops_max" : 299534,\n'
                                             '        "iops_mean" : '
                                             '297256.000000,\n'
                                             '        "iops_stddev" : '
                                             '932.970030,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 59.700000,\n'
                                             '      "sys_cpu" : 40.271429,\n'
                                             '      "ctx" : 10,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.100000,\n'
                                             '        "2" : 99.989808,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.010000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.010000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 2.073292,\n'
                                             '        "10" : 95.884665,\n'
                                             '        "20" : 1.778440,\n'
                                             '        "50" : 0.241055,\n'
                                             '        "100" : 0.020096,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 2,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'libaio-bdev_aio: (groupid=0, '
                                             'jobs=1): err= 0: pid=7997: Wed '
                                             'Oct 26 15:13:25 2022\n'
                                             '  read: IOPS=297k, BW=145MiB/s '
                                             '(152MB/s)(1016MiB/7001msec)\n'
                                             '    slat (nsec): min=1663, '
                                             'max=18003, avg=1897.38, '
                                             'stdev=130.28\n'
                                             '    clat (nsec): min=250, '
                                             'max=117160, avg=4608.01, '
                                             'stdev=1767.88\n'
                                             '     lat (usec): min=5, max=119, '
                                             'avg= 6.51, stdev= 1.77\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 3952],  '
                                             '5.00th=[ 4192], 10.00th=[ 4256], '
                                             '20.00th=[ 4256],\n'
                                             '     | 30.00th=[ 4320], '
                                             '40.00th=[ 4320], 50.00th=[ '
                                             '4384], 60.00th=[ 4384],\n'
                                             '     | 70.00th=[ 4448], '
                                             '80.00th=[ 4448], 90.00th=[ '
                                             '4512], 95.00th=[ 4704],\n'
                                             '     | 99.00th=[12608], '
                                             '99.50th=[17536], '
                                             '99.90th=[21888], '
                                             '99.95th=[22400],\n'
                                             '     | 99.99th=[61184]\n'
                                             '   bw (  KiB/s): min=147892, '
                                             'max=149767, per=100.00%, '
                                             'avg=148627.93, stdev=466.57, '
                                             'samples=14\n'
                                             '   iops        : min=295785, '
                                             'max=299534, avg=297256.00, '
                                             'stdev=932.97, samples=14\n'
                                             '  lat (nsec)   : 500=0.01%, '
                                             '1000=0.01%\n'
                                             '  lat (usec)   : 4=2.07%, '
                                             '10=95.88%, 20=1.78%, 50=0.24%, '
                                             '100=0.02%\n'
                                             '  lat (usec)   : 250=0.01%\n'
                                             '  cpu          : usr=59.70%, '
                                             'sys=40.27%, ctx=10, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.1%, '
                                             '2=100.0%, 4=0.0%, 8=0.0%, '
                                             '16=0.0%, 32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=2080025,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=2\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=145MiB/s (152MB/s), '
                                             '145MiB/s-145MiB/s '
                                             '(152MB/s-152MB/s), io=1016MiB '
                                             '(1065MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_180.output'),
                                   'state': {'begin': 1666782915.1175997,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_2_libaio-bdev_aio_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8616042137145996,
                                             'end': 1666782915.979204,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_180.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_180.state')},
                       'cmd_181': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_512_2_libaio-bdev_xnvme_0.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_181.output'),
                                   'state': {'begin': 1666782918.1243794,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_2_libaio-bdev_xnvme_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8209547996520996,
                                             'end': 1666782918.9453342,
                                             'err': 1,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_181.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_181.state')},
                       'cmd_182': {'output': 'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_182.output'),
                                   'state': {'begin': 1666782918.947441,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="libaio-bdev_xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="2" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="spdk_bdev" '
                                                    '--spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" '
                                                    '--filename="Nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_2_libaio-bdev_xnvme_0.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.447400093078613,
                                             'end': 1666782930.3948412,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_182.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_182.state')},
                       'cmd_183': {'output': 'libaio-bdev_xnvme: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=spdk_bdev, iodepth=2\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666790020,\n'
                                             '  "timestamp_ms" : '
                                             '1666790020382,\n'
                                             '  "time" : "Wed Oct 26 15:13:40 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"libaio-bdev_xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"libaio-bdev_xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "2",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"spdk_bdev",\n'
                                             '        "spdk_conf" : '
                                             '"/tmp/spdk_bdev_xnvme_libaio.conf",\n'
                                             '        "filename" : "Nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '1019141632,\n'
                                             '        "io_kbytes" : 995255,\n'
                                             '        "bw_bytes" : 145570865,\n'
                                             '        "bw" : 142159,\n'
                                             '        "iops" : 284317.954578,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 1990510,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1693,\n'
                                             '          "max" : 23505,\n'
                                             '          "mean" : 1928.243363,\n'
                                             '          "stddev" : '
                                             '145.937912,\n'
                                             '          "N" : 1990510\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 611,\n'
                                             '          "max" : 114164,\n'
                                             '          "mean" : 4850.434752,\n'
                                             '          "stddev" : '
                                             '1749.968240,\n'
                                             '          "N" : 1990511,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 4256,\n'
                                             '            "5.000000" : 4384,\n'
                                             '            "10.000000" : 4384,\n'
                                             '            "20.000000" : 4512,\n'
                                             '            "30.000000" : 4576,\n'
                                             '            "40.000000" : 4576,\n'
                                             '            "50.000000" : 4640,\n'
                                             '            "60.000000" : 4640,\n'
                                             '            "70.000000" : 4640,\n'
                                             '            "80.000000" : 4704,\n'
                                             '            "90.000000" : 4768,\n'
                                             '            "95.000000" : 4896,\n'
                                             '            "99.000000" : '
                                             '12736,\n'
                                             '            "99.500000" : '
                                             '18048,\n'
                                             '            "99.900000" : '
                                             '21888,\n'
                                             '            "99.950000" : '
                                             '22144,\n'
                                             '            "99.990000" : 60672\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 5460,\n'
                                             '          "max" : 116057,\n'
                                             '          "mean" : 6778.678102,\n'
                                             '          "stddev" : '
                                             '1753.594753,\n'
                                             '          "N" : 1990511\n'
                                             '        },\n'
                                             '        "bw_min" : 141762,\n'
                                             '        "bw_max" : 142974,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '142184.785714,\n'
                                             '        "bw_dev" : 341.212809,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 283524,\n'
                                             '        "iops_max" : 285949,\n'
                                             '        "iops_mean" : '
                                             '284369.642857,\n'
                                             '        "iops_stddev" : '
                                             '682.603569,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 56.500000,\n'
                                             '      "sys_cpu" : 43.471429,\n'
                                             '      "ctx" : 7,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.100000,\n'
                                             '        "2" : 99.977795,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.010000,\n'
                                             '        "1000" : 0.010000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.017684,\n'
                                             '        "10" : 97.705714,\n'
                                             '        "20" : 1.999236,\n'
                                             '        "50" : 0.251091,\n'
                                             '        "100" : 0.019744,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 2,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'libaio-bdev_xnvme: (groupid=0, '
                                             'jobs=1): err= 0: pid=8054: Wed '
                                             'Oct 26 15:13:40 2022\n'
                                             '  read: IOPS=284k, BW=139MiB/s '
                                             '(146MB/s)(972MiB/7001msec)\n'
                                             '    slat (nsec): min=1693, '
                                             'max=23505, avg=1928.24, '
                                             'stdev=145.94\n'
                                             '    clat (nsec): min=611, '
                                             'max=114164, avg=4850.43, '
                                             'stdev=1749.97\n'
                                             '     lat (usec): min=5, max=116, '
                                             'avg= 6.78, stdev= 1.75\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 4256],  '
                                             '5.00th=[ 4384], 10.00th=[ 4384], '
                                             '20.00th=[ 4512],\n'
                                             '     | 30.00th=[ 4576], '
                                             '40.00th=[ 4576], 50.00th=[ '
                                             '4640], 60.00th=[ 4640],\n'
                                             '     | 70.00th=[ 4640], '
                                             '80.00th=[ 4704], 90.00th=[ '
                                             '4768], 95.00th=[ 4896],\n'
                                             '     | 99.00th=[12736], '
                                             '99.50th=[18048], '
                                             '99.90th=[21888], '
                                             '99.95th=[22144],\n'
                                             '     | 99.99th=[60672]\n'
                                             '   bw (  KiB/s): min=141762, '
                                             'max=142974, per=100.00%, '
                                             'avg=142184.79, stdev=341.21, '
                                             'samples=14\n'
                                             '   iops        : min=283524, '
                                             'max=285949, avg=284369.64, '
                                             'stdev=682.60, samples=14\n'
                                             '  lat (nsec)   : 750=0.01%, '
                                             '1000=0.01%\n'
                                             '  lat (usec)   : 4=0.02%, '
                                             '10=97.71%, 20=2.00%, 50=0.25%, '
                                             '100=0.02%\n'
                                             '  lat (usec)   : 250=0.01%\n'
                                             '  cpu          : usr=56.50%, '
                                             'sys=43.47%, ctx=7, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.1%, '
                                             '2=100.0%, 4=0.0%, 8=0.0%, '
                                             '16=0.0%, 32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=1990510,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=2\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=139MiB/s (146MB/s), '
                                             '139MiB/s-139MiB/s '
                                             '(146MB/s-146MB/s), io=972MiB '
                                             '(1019MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_183.output'),
                                   'state': {'begin': 1666782930.3976998,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_2_libaio-bdev_xnvme_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8358349800109863,
                                             'end': 1666782931.2335348,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_183.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_183.state')},
                       'cmd_184': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_512_2_libaio-bdev_xnvme_1.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_184.output'),
                                   'state': {'begin': 1666782933.2798376,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_2_libaio-bdev_xnvme_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7966079711914062,
                                             'end': 1666782934.0764456,
                                             'err': 1,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_184.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_184.state')},
                       'cmd_185': {'output': 'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_185.output'),
                                   'state': {'begin': 1666782934.0783417,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="libaio-bdev_xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="2" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="spdk_bdev" '
                                                    '--spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" '
                                                    '--filename="Nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_2_libaio-bdev_xnvme_1.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.518922328948975,
                                             'end': 1666782945.597264,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_185.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_185.state')},
                       'cmd_186': {'output': 'libaio-bdev_xnvme: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=spdk_bdev, iodepth=2\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666790035,\n'
                                             '  "timestamp_ms" : '
                                             '1666790035564,\n'
                                             '  "time" : "Wed Oct 26 15:13:55 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"libaio-bdev_xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"libaio-bdev_xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "2",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"spdk_bdev",\n'
                                             '        "spdk_conf" : '
                                             '"/tmp/spdk_bdev_xnvme_libaio.conf",\n'
                                             '        "filename" : "Nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '1025831424,\n'
                                             '        "io_kbytes" : 1001788,\n'
                                             '        "bw_bytes" : 146526413,\n'
                                             '        "bw" : 143092,\n'
                                             '        "iops" : 286184.259392,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 2003576,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1713,\n'
                                             '          "max" : 22923,\n'
                                             '          "mean" : 1939.483181,\n'
                                             '          "stddev" : '
                                             '128.999116,\n'
                                             '          "N" : 2003576\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 652,\n'
                                             '          "max" : 114214,\n'
                                             '          "mean" : 4820.917562,\n'
                                             '          "stddev" : '
                                             '1748.341345,\n'
                                             '          "N" : 2003577,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 4256,\n'
                                             '            "5.000000" : 4384,\n'
                                             '            "10.000000" : 4448,\n'
                                             '            "20.000000" : 4512,\n'
                                             '            "30.000000" : 4512,\n'
                                             '            "40.000000" : 4576,\n'
                                             '            "50.000000" : 4576,\n'
                                             '            "60.000000" : 4576,\n'
                                             '            "70.000000" : 4640,\n'
                                             '            "80.000000" : 4640,\n'
                                             '            "90.000000" : 4768,\n'
                                             '            "95.000000" : 4896,\n'
                                             '            "99.000000" : '
                                             '12736,\n'
                                             '            "99.500000" : '
                                             '18048,\n'
                                             '            "99.900000" : '
                                             '21888,\n'
                                             '            "99.950000" : '
                                             '22144,\n'
                                             '            "99.990000" : 61184\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 5680,\n'
                                             '          "max" : 116118,\n'
                                             '          "mean" : 6760.400710,\n'
                                             '          "stddev" : '
                                             '1752.206198,\n'
                                             '          "N" : 2003577\n'
                                             '        },\n'
                                             '        "bw_min" : 142671,\n'
                                             '        "bw_max" : 143940,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '143196.692308,\n'
                                             '        "bw_dev" : 456.645629,\n'
                                             '        "bw_samples" : 13,\n'
                                             '        "iops_min" : 285342,\n'
                                             '        "iops_max" : 287880,\n'
                                             '        "iops_mean" : '
                                             '286393.538462,\n'
                                             '        "iops_stddev" : '
                                             '913.368638,\n'
                                             '        "iops_samples" : 13\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 55.757143,\n'
                                             '      "sys_cpu" : 44.228571,\n'
                                             '      "ctx" : 9,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.100000,\n'
                                             '        "2" : 99.987323,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.010000,\n'
                                             '        "1000" : 0.010000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.010000,\n'
                                             '        "4" : 0.023558,\n'
                                             '        "10" : 97.738543,\n'
                                             '        "20" : 1.962641,\n'
                                             '        "50" : 0.251900,\n'
                                             '        "100" : 0.019815,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 2,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'libaio-bdev_xnvme: (groupid=0, '
                                             'jobs=1): err= 0: pid=8112: Wed '
                                             'Oct 26 15:13:55 2022\n'
                                             '  read: IOPS=286k, BW=140MiB/s '
                                             '(147MB/s)(978MiB/7001msec)\n'
                                             '    slat (nsec): min=1713, '
                                             'max=22923, avg=1939.48, '
                                             'stdev=129.00\n'
                                             '    clat (nsec): min=652, '
                                             'max=114214, avg=4820.92, '
                                             'stdev=1748.34\n'
                                             '     lat (usec): min=5, max=116, '
                                             'avg= 6.76, stdev= 1.75\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 4256],  '
                                             '5.00th=[ 4384], 10.00th=[ 4448], '
                                             '20.00th=[ 4512],\n'
                                             '     | 30.00th=[ 4512], '
                                             '40.00th=[ 4576], 50.00th=[ '
                                             '4576], 60.00th=[ 4576],\n'
                                             '     | 70.00th=[ 4640], '
                                             '80.00th=[ 4640], 90.00th=[ '
                                             '4768], 95.00th=[ 4896],\n'
                                             '     | 99.00th=[12736], '
                                             '99.50th=[18048], '
                                             '99.90th=[21888], '
                                             '99.95th=[22144],\n'
                                             '     | 99.99th=[61184]\n'
                                             '   bw (  KiB/s): min=142671, '
                                             'max=143940, per=100.00%, '
                                             'avg=143196.69, stdev=456.65, '
                                             'samples=13\n'
                                             '   iops        : min=285342, '
                                             'max=287880, avg=286393.54, '
                                             'stdev=913.37, samples=13\n'
                                             '  lat (nsec)   : 750=0.01%, '
                                             '1000=0.01%\n'
                                             '  lat (usec)   : 2=0.01%, '
                                             '4=0.02%, 10=97.74%, 20=1.96%, '
                                             '50=0.25%\n'
                                             '  lat (usec)   : 100=0.02%, '
                                             '250=0.01%\n'
                                             '  cpu          : usr=55.76%, '
                                             'sys=44.23%, ctx=9, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.1%, '
                                             '2=100.0%, 4=0.0%, 8=0.0%, '
                                             '16=0.0%, 32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=2003576,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=2\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=140MiB/s (147MB/s), '
                                             '140MiB/s-140MiB/s '
                                             '(147MB/s-147MB/s), io=978MiB '
                                             '(1026MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_186.output'),
                                   'state': {'begin': 1666782945.599773,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_2_libaio-bdev_xnvme_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.804248571395874,
                                             'end': 1666782946.4040215,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_186.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_186.state')},
                       'cmd_187': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_512_2_libaio-bdev_xnvme_2.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_187.output'),
                                   'state': {'begin': 1666782948.5101879,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_2_libaio-bdev_xnvme_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7871615886688232,
                                             'end': 1666782949.2973495,
                                             'err': 1,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_187.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_187.state')},
                       'cmd_188': {'output': 'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_188.output'),
                                   'state': {'begin': 1666782949.2993407,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="libaio-bdev_xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="2" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="spdk_bdev" '
                                                    '--spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" '
                                                    '--filename="Nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_2_libaio-bdev_xnvme_2.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.528333902359009,
                                             'end': 1666782960.8276746,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_188.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_188.state')},
                       'cmd_189': {'output': 'libaio-bdev_xnvme: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=spdk_bdev, iodepth=2\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666790050,\n'
                                             '  "timestamp_ms" : '
                                             '1666790050783,\n'
                                             '  "time" : "Wed Oct 26 15:14:10 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"libaio-bdev_xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"libaio-bdev_xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "2",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"spdk_bdev",\n'
                                             '        "spdk_conf" : '
                                             '"/tmp/spdk_bdev_xnvme_libaio.conf",\n'
                                             '        "filename" : "Nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '1025968128,\n'
                                             '        "io_kbytes" : 1001922,\n'
                                             '        "bw_bytes" : 146566875,\n'
                                             '        "bw" : 143131,\n'
                                             '        "iops" : 286263.142857,\n'
                                             '        "runtime" : 7000,\n'
                                             '        "total_ios" : 2003842,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1683,\n'
                                             '          "max" : 22953,\n'
                                             '          "mean" : 1917.783290,\n'
                                             '          "stddev" : '
                                             '143.998960,\n'
                                             '          "N" : 2003843\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 651,\n'
                                             '          "max" : 119825,\n'
                                             '          "mean" : 4827.615465,\n'
                                             '          "stddev" : '
                                             '1728.532459,\n'
                                             '          "N" : 2003842,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 4256,\n'
                                             '            "5.000000" : 4384,\n'
                                             '            "10.000000" : 4384,\n'
                                             '            "20.000000" : 4448,\n'
                                             '            "30.000000" : 4512,\n'
                                             '            "40.000000" : 4576,\n'
                                             '            "50.000000" : 4640,\n'
                                             '            "60.000000" : 4640,\n'
                                             '            "70.000000" : 4640,\n'
                                             '            "80.000000" : 4704,\n'
                                             '            "90.000000" : 4768,\n'
                                             '            "95.000000" : 4960,\n'
                                             '            "99.000000" : '
                                             '12736,\n'
                                             '            "99.500000" : '
                                             '18048,\n'
                                             '            "99.900000" : '
                                             '21888,\n'
                                             '            "99.950000" : '
                                             '22144,\n'
                                             '            "99.990000" : 59648\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 5410,\n'
                                             '          "max" : 121838,\n'
                                             '          "mean" : 6745.398757,\n'
                                             '          "stddev" : '
                                             '1732.651265,\n'
                                             '          "N" : 2003842\n'
                                             '        },\n'
                                             '        "bw_min" : 142876,\n'
                                             '        "bw_max" : 143492,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '143137.428571,\n'
                                             '        "bw_dev" : 166.771476,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 285752,\n'
                                             '        "iops_max" : 286984,\n'
                                             '        "iops_mean" : '
                                             '286274.928571,\n'
                                             '        "iops_stddev" : '
                                             '333.438142,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 55.185714,\n'
                                             '      "sys_cpu" : 44.785714,\n'
                                             '      "ctx" : 7,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.100000,\n'
                                             '        "2" : 99.978691,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.010000,\n'
                                             '        "1000" : 0.010000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.016418,\n'
                                             '        "10" : 97.727366,\n'
                                             '        "20" : 1.985785,\n'
                                             '        "50" : 0.247275,\n'
                                             '        "100" : 0.017267,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 2,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'libaio-bdev_xnvme: (groupid=0, '
                                             'jobs=1): err= 0: pid=8172: Wed '
                                             'Oct 26 15:14:10 2022\n'
                                             '  read: IOPS=286k, BW=140MiB/s '
                                             '(147MB/s)(978MiB/7000msec)\n'
                                             '    slat (nsec): min=1683, '
                                             'max=22953, avg=1917.78, '
                                             'stdev=144.00\n'
                                             '    clat (nsec): min=651, '
                                             'max=119825, avg=4827.62, '
                                             'stdev=1728.53\n'
                                             '     lat (usec): min=5, max=121, '
                                             'avg= 6.75, stdev= 1.73\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 4256],  '
                                             '5.00th=[ 4384], 10.00th=[ 4384], '
                                             '20.00th=[ 4448],\n'
                                             '     | 30.00th=[ 4512], '
                                             '40.00th=[ 4576], 50.00th=[ '
                                             '4640], 60.00th=[ 4640],\n'
                                             '     | 70.00th=[ 4640], '
                                             '80.00th=[ 4704], 90.00th=[ '
                                             '4768], 95.00th=[ 4960],\n'
                                             '     | 99.00th=[12736], '
                                             '99.50th=[18048], '
                                             '99.90th=[21888], '
                                             '99.95th=[22144],\n'
                                             '     | 99.99th=[59648]\n'
                                             '   bw (  KiB/s): min=142876, '
                                             'max=143492, per=100.00%, '
                                             'avg=143137.43, stdev=166.77, '
                                             'samples=14\n'
                                             '   iops        : min=285752, '
                                             'max=286984, avg=286274.93, '
                                             'stdev=333.44, samples=14\n'
                                             '  lat (nsec)   : 750=0.01%, '
                                             '1000=0.01%\n'
                                             '  lat (usec)   : 4=0.02%, '
                                             '10=97.73%, 20=1.99%, 50=0.25%, '
                                             '100=0.02%\n'
                                             '  lat (usec)   : 250=0.01%\n'
                                             '  cpu          : usr=55.19%, '
                                             'sys=44.79%, ctx=7, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.1%, '
                                             '2=100.0%, 4=0.0%, 8=0.0%, '
                                             '16=0.0%, 32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=2003842,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=2\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=140MiB/s (147MB/s), '
                                             '140MiB/s-140MiB/s '
                                             '(147MB/s-147MB/s), io=978MiB '
                                             '(1026MB), run=7000-7000msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_189.output'),
                                   'state': {'begin': 1666782960.8303506,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_2_libaio-bdev_xnvme_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8077137470245361,
                                             'end': 1666782961.6380644,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_189.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_189.state')},
                       'cmd_19': {'output': 'rm: cannot remove '
                                            "'/tmp/fio-output_512_1_io_uring_cmd-bdev_xnvme_0.txt': "
                                            'No such file or directory\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_19.output'),
                                  'state': {'begin': 1666782130.4222083,
                                            'cmd': 'rm '
                                                   '/tmp/fio-output_512_1_io_uring_cmd-bdev_xnvme_0.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.864431619644165,
                                            'end': 1666782131.28664,
                                            'err': 1,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_19.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_19.state')},
                       'cmd_190': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_512_2_libaio-xnvme_0.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_190.output'),
                                   'state': {'begin': 1666782962.6391451,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_2_libaio-xnvme_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8529889583587646,
                                             'end': 1666782963.492134,
                                             'err': 1,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_190.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_190.state')},
                       'cmd_191': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_191.output'),
                                   'state': {'begin': 1666782963.4948034,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="libaio-xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="2" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
                                                    '--xnvme_be="linux" '
                                                    '--xnvme_async="libaio" '
                                                    '--xnvme_sync="nvme" '
                                                    '--xnvme_admin="nvme" '
                                                    '--xnvme_dev_nsid="1" '
                                                    '--filename="/dev/nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_2_libaio-xnvme_0.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.136729001998901,
                                             'end': 1666782974.6315324,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_191.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_191.state')},
                       'cmd_192': {'output': 'libaio-xnvme: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=xnvme, iodepth=2\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666790064,\n'
                                             '  "timestamp_ms" : '
                                             '1666790064713,\n'
                                             '  "time" : "Wed Oct 26 15:14:24 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"libaio-xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"libaio-xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "2",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
                                             '        "xnvme_be" : "linux",\n'
                                             '        "xnvme_async" : '
                                             '"libaio",\n'
                                             '        "xnvme_sync" : "nvme",\n'
                                             '        "xnvme_admin" : "nvme",\n'
                                             '        "xnvme_dev_nsid" : "1",\n'
                                             '        "filename" : '
                                             '"/dev/nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : 840821760,\n'
                                             '        "io_kbytes" : 821115,\n'
                                             '        "bw_bytes" : 120100237,\n'
                                             '        "bw" : 117285,\n'
                                             '        "iops" : 234570.489930,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 1642228,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1503,\n'
                                             '          "max" : 27501,\n'
                                             '          "mean" : 1736.627195,\n'
                                             '          "stddev" : '
                                             '156.192241,\n'
                                             '          "N" : 1642229\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 2836,\n'
                                             '          "max" : 121638,\n'
                                             '          "mean" : 6548.460939,\n'
                                             '          "stddev" : '
                                             '1775.773390,\n'
                                             '          "N" : 1642229,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 5920,\n'
                                             '            "5.000000" : 5920,\n'
                                             '            "10.000000" : 5984,\n'
                                             '            "20.000000" : 6048,\n'
                                             '            "30.000000" : 6112,\n'
                                             '            "40.000000" : 6176,\n'
                                             '            "50.000000" : 6240,\n'
                                             '            "60.000000" : 6304,\n'
                                             '            "70.000000" : 6368,\n'
                                             '            "80.000000" : 6432,\n'
                                             '            "90.000000" : 7136,\n'
                                             '            "95.000000" : 7584,\n'
                                             '            "99.000000" : '
                                             '15040,\n'
                                             '            "99.500000" : '
                                             '18560,\n'
                                             '            "99.900000" : '
                                             '25984,\n'
                                             '            "99.950000" : '
                                             '26496,\n'
                                             '            "99.990000" : 62208\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 6603,\n'
                                             '          "max" : 123481,\n'
                                             '          "mean" : 8285.088133,\n'
                                             '          "stddev" : '
                                             '1774.881999,\n'
                                             '          "N" : 1642229\n'
                                             '        },\n'
                                             '        "bw_min" : 116619,\n'
                                             '        "bw_max" : 117728,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '117305.857143,\n'
                                             '        "bw_dev" : 299.454853,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 233238,\n'
                                             '        "iops_max" : 235456,\n'
                                             '        "iops_mean" : '
                                             '234611.928571,\n'
                                             '        "iops_stddev" : '
                                             '598.818126,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 19.800000,\n'
                                             '      "sys_cpu" : 57.285714,\n'
                                             '      "ctx" : 824948,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 100.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 97.830691,\n'
                                             '        "20" : 1.792078,\n'
                                             '        "50" : 0.349099,\n'
                                             '        "100" : 0.020764,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 2,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'libaio-xnvme: (groupid=0, '
                                             'jobs=1): err= 0: pid=8219: Wed '
                                             'Oct 26 15:14:24 2022\n'
                                             '  read: IOPS=235k, BW=115MiB/s '
                                             '(120MB/s)(802MiB/7001msec)\n'
                                             '    slat (nsec): min=1503, '
                                             'max=27501, avg=1736.63, '
                                             'stdev=156.19\n'
                                             '    clat (usec): min=2, max=121, '
                                             'avg= 6.55, stdev= 1.78\n'
                                             '     lat (usec): min=6, max=123, '
                                             'avg= 8.29, stdev= 1.77\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 5920],  '
                                             '5.00th=[ 5920], 10.00th=[ 5984], '
                                             '20.00th=[ 6048],\n'
                                             '     | 30.00th=[ 6112], '
                                             '40.00th=[ 6176], 50.00th=[ '
                                             '6240], 60.00th=[ 6304],\n'
                                             '     | 70.00th=[ 6368], '
                                             '80.00th=[ 6432], 90.00th=[ '
                                             '7136], 95.00th=[ 7584],\n'
                                             '     | 99.00th=[15040], '
                                             '99.50th=[18560], '
                                             '99.90th=[25984], '
                                             '99.95th=[26496],\n'
                                             '     | 99.99th=[62208]\n'
                                             '   bw (  KiB/s): min=116619, '
                                             'max=117728, per=100.00%, '
                                             'avg=117305.86, stdev=299.45, '
                                             'samples=14\n'
                                             '   iops        : min=233238, '
                                             'max=235456, avg=234611.93, '
                                             'stdev=598.82, samples=14\n'
                                             '  lat (usec)   : 4=0.01%, '
                                             '10=97.83%, 20=1.79%, 50=0.35%, '
                                             '100=0.02%\n'
                                             '  lat (usec)   : 250=0.01%\n'
                                             '  cpu          : usr=19.80%, '
                                             'sys=57.29%, ctx=824948, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, '
                                             '2=100.0%, 4=0.0%, 8=0.0%, '
                                             '16=0.0%, 32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=1642228,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=2\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=115MiB/s (120MB/s), '
                                             '115MiB/s-115MiB/s '
                                             '(120MB/s-120MB/s), io=802MiB '
                                             '(841MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_192.output'),
                                   'state': {'begin': 1666782974.6349154,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_2_libaio-xnvme_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8530197143554688,
                                             'end': 1666782975.487935,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_192.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_192.state')},
                       'cmd_193': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_512_2_libaio-xnvme_1.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_193.output'),
                                   'state': {'begin': 1666782976.5959628,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_2_libaio-xnvme_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8007280826568604,
                                             'end': 1666782977.3966908,
                                             'err': 1,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_193.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_193.state')},
                       'cmd_194': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_194.output'),
                                   'state': {'begin': 1666782977.398984,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="libaio-xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="2" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
                                                    '--xnvme_be="linux" '
                                                    '--xnvme_async="libaio" '
                                                    '--xnvme_sync="nvme" '
                                                    '--xnvme_admin="nvme" '
                                                    '--xnvme_dev_nsid="1" '
                                                    '--filename="/dev/nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_2_libaio-xnvme_1.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.133901357650757,
                                             'end': 1666782988.5328853,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_194.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_194.state')},
                       'cmd_195': {'output': 'libaio-xnvme: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=xnvme, iodepth=2\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666790078,\n'
                                             '  "timestamp_ms" : '
                                             '1666790078614,\n'
                                             '  "time" : "Wed Oct 26 15:14:38 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"libaio-xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"libaio-xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "2",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
                                             '        "xnvme_be" : "linux",\n'
                                             '        "xnvme_async" : '
                                             '"libaio",\n'
                                             '        "xnvme_sync" : "nvme",\n'
                                             '        "xnvme_admin" : "nvme",\n'
                                             '        "xnvme_dev_nsid" : "1",\n'
                                             '        "filename" : '
                                             '"/dev/nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : 837950976,\n'
                                             '        "io_kbytes" : 818311,\n'
                                             '        "bw_bytes" : 119690183,\n'
                                             '        "bw" : 116884,\n'
                                             '        "iops" : 233769.747179,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 1636622,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1513,\n'
                                             '          "max" : 34655,\n'
                                             '          "mean" : 1732.315724,\n'
                                             '          "stddev" : '
                                             '166.086574,\n'
                                             '          "N" : 1636622\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 2925,\n'
                                             '          "max" : 119975,\n'
                                             '          "mean" : 6582.195259,\n'
                                             '          "stddev" : '
                                             '1784.605938,\n'
                                             '          "N" : 1636623,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 5920,\n'
                                             '            "5.000000" : 5984,\n'
                                             '            "10.000000" : 5984,\n'
                                             '            "20.000000" : 6048,\n'
                                             '            "30.000000" : 6112,\n'
                                             '            "40.000000" : 6176,\n'
                                             '            "50.000000" : 6304,\n'
                                             '            "60.000000" : 6368,\n'
                                             '            "70.000000" : 6432,\n'
                                             '            "80.000000" : 6496,\n'
                                             '            "90.000000" : 7072,\n'
                                             '            "95.000000" : 7584,\n'
                                             '            "99.000000" : '
                                             '15296,\n'
                                             '            "99.500000" : '
                                             '18560,\n'
                                             '            "99.900000" : '
                                             '25984,\n'
                                             '            "99.950000" : '
                                             '26496,\n'
                                             '            "99.990000" : 62720\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 5931,\n'
                                             '          "max" : 121779,\n'
                                             '          "mean" : 8314.510898,\n'
                                             '          "stddev" : '
                                             '1783.242623,\n'
                                             '          "N" : 1636623\n'
                                             '        },\n'
                                             '        "bw_min" : 116388,\n'
                                             '        "bw_max" : 117454,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '116959.071429,\n'
                                             '        "bw_dev" : 338.761878,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 232776,\n'
                                             '        "iops_max" : 234908,\n'
                                             '        "iops_mean" : '
                                             '233918.357143,\n'
                                             '        "iops_stddev" : '
                                             '677.455262,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 20.128571,\n'
                                             '      "sys_cpu" : 56.842857,\n'
                                             '      "ctx" : 822028,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 100.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 97.817945,\n'
                                             '        "20" : 1.790823,\n'
                                             '        "50" : 0.360560,\n'
                                             '        "100" : 0.022241,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 2,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'libaio-xnvme: (groupid=0, '
                                             'jobs=1): err= 0: pid=8268: Wed '
                                             'Oct 26 15:14:38 2022\n'
                                             '  read: IOPS=234k, BW=114MiB/s '
                                             '(120MB/s)(799MiB/7001msec)\n'
                                             '    slat (nsec): min=1513, '
                                             'max=34655, avg=1732.32, '
                                             'stdev=166.09\n'
                                             '    clat (usec): min=2, max=119, '
                                             'avg= 6.58, stdev= 1.78\n'
                                             '     lat (usec): min=5, max=121, '
                                             'avg= 8.31, stdev= 1.78\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 5920],  '
                                             '5.00th=[ 5984], 10.00th=[ 5984], '
                                             '20.00th=[ 6048],\n'
                                             '     | 30.00th=[ 6112], '
                                             '40.00th=[ 6176], 50.00th=[ '
                                             '6304], 60.00th=[ 6368],\n'
                                             '     | 70.00th=[ 6432], '
                                             '80.00th=[ 6496], 90.00th=[ '
                                             '7072], 95.00th=[ 7584],\n'
                                             '     | 99.00th=[15296], '
                                             '99.50th=[18560], '
                                             '99.90th=[25984], '
                                             '99.95th=[26496],\n'
                                             '     | 99.99th=[62720]\n'
                                             '   bw (  KiB/s): min=116388, '
                                             'max=117454, per=100.00%, '
                                             'avg=116959.07, stdev=338.76, '
                                             'samples=14\n'
                                             '   iops        : min=232776, '
                                             'max=234908, avg=233918.36, '
                                             'stdev=677.46, samples=14\n'
                                             '  lat (usec)   : 4=0.01%, '
                                             '10=97.82%, 20=1.79%, 50=0.36%, '
                                             '100=0.02%\n'
                                             '  lat (usec)   : 250=0.01%\n'
                                             '  cpu          : usr=20.13%, '
                                             'sys=56.84%, ctx=822028, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, '
                                             '2=100.0%, 4=0.0%, 8=0.0%, '
                                             '16=0.0%, 32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=1636622,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=2\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=114MiB/s (120MB/s), '
                                             '114MiB/s-114MiB/s '
                                             '(120MB/s-120MB/s), io=799MiB '
                                             '(838MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_195.output'),
                                   'state': {'begin': 1666782988.5356045,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_2_libaio-xnvme_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8266096115112305,
                                             'end': 1666782989.362214,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_195.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_195.state')},
                       'cmd_196': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_512_2_libaio-xnvme_2.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_196.output'),
                                   'state': {'begin': 1666782990.3637896,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_2_libaio-xnvme_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8320932388305664,
                                             'end': 1666782991.1958828,
                                             'err': 1,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_196.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_196.state')},
                       'cmd_197': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_197.output'),
                                   'state': {'begin': 1666782991.1985276,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="libaio-xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="2" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
                                                    '--xnvme_be="linux" '
                                                    '--xnvme_async="libaio" '
                                                    '--xnvme_sync="nvme" '
                                                    '--xnvme_admin="nvme" '
                                                    '--xnvme_dev_nsid="1" '
                                                    '--filename="/dev/nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_2_libaio-xnvme_2.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.097421169281006,
                                             'end': 1666783002.2959487,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_197.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_197.state')},
                       'cmd_198': {'output': 'libaio-xnvme: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=xnvme, iodepth=2\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666790092,\n'
                                             '  "timestamp_ms" : '
                                             '1666790092370,\n'
                                             '  "time" : "Wed Oct 26 15:14:52 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"libaio-xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"libaio-xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "2",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
                                             '        "xnvme_be" : "linux",\n'
                                             '        "xnvme_async" : '
                                             '"libaio",\n'
                                             '        "xnvme_sync" : "nvme",\n'
                                             '        "xnvme_admin" : "nvme",\n'
                                             '        "xnvme_dev_nsid" : "1",\n'
                                             '        "filename" : '
                                             '"/dev/nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : 836393472,\n'
                                             '        "io_kbytes" : 816790,\n'
                                             '        "bw_bytes" : 119467714,\n'
                                             '        "bw" : 116667,\n'
                                             '        "iops" : 233335.237823,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 1633580,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1512,\n'
                                             '          "max" : 53531,\n'
                                             '          "mean" : 1744.740057,\n'
                                             '          "stddev" : '
                                             '169.297769,\n'
                                             '          "N" : 1633580\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 2875,\n'
                                             '          "max" : 119985,\n'
                                             '          "mean" : 6578.702695,\n'
                                             '          "stddev" : '
                                             '1794.962820,\n'
                                             '          "N" : 1633581,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 5920,\n'
                                             '            "5.000000" : 5984,\n'
                                             '            "10.000000" : 5984,\n'
                                             '            "20.000000" : 6048,\n'
                                             '            "30.000000" : 6112,\n'
                                             '            "40.000000" : 6176,\n'
                                             '            "50.000000" : 6304,\n'
                                             '            "60.000000" : 6304,\n'
                                             '            "70.000000" : 6368,\n'
                                             '            "80.000000" : 6496,\n'
                                             '            "90.000000" : 7136,\n'
                                             '            "95.000000" : 7584,\n'
                                             '            "99.000000" : '
                                             '15296,\n'
                                             '            "99.500000" : '
                                             '18560,\n'
                                             '            "99.900000" : '
                                             '25984,\n'
                                             '            "99.950000" : '
                                             '26496,\n'
                                             '            "99.990000" : 62720\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 6793,\n'
                                             '          "max" : 121838,\n'
                                             '          "mean" : 8323.442636,\n'
                                             '          "stddev" : '
                                             '1795.815444,\n'
                                             '          "N" : 1633581\n'
                                             '        },\n'
                                             '        "bw_min" : 115883,\n'
                                             '        "bw_max" : 117515,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '116731.142857,\n'
                                             '        "bw_dev" : 532.190951,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 231766,\n'
                                             '        "iops_max" : 235030,\n'
                                             '        "iops_mean" : '
                                             '233462.428571,\n'
                                             '        "iops_stddev" : '
                                             '1064.482230,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 20.028571,\n'
                                             '      "sys_cpu" : 57.300000,\n'
                                             '      "ctx" : 820575,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 100.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 97.820186,\n'
                                             '        "20" : 1.795872,\n'
                                             '        "50" : 0.354130,\n'
                                             '        "100" : 0.022833,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 2,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'libaio-xnvme: (groupid=0, '
                                             'jobs=1): err= 0: pid=8316: Wed '
                                             'Oct 26 15:14:52 2022\n'
                                             '  read: IOPS=233k, BW=114MiB/s '
                                             '(119MB/s)(798MiB/7001msec)\n'
                                             '    slat (nsec): min=1512, '
                                             'max=53531, avg=1744.74, '
                                             'stdev=169.30\n'
                                             '    clat (usec): min=2, max=119, '
                                             'avg= 6.58, stdev= 1.79\n'
                                             '     lat (usec): min=6, max=121, '
                                             'avg= 8.32, stdev= 1.80\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 5920],  '
                                             '5.00th=[ 5984], 10.00th=[ 5984], '
                                             '20.00th=[ 6048],\n'
                                             '     | 30.00th=[ 6112], '
                                             '40.00th=[ 6176], 50.00th=[ '
                                             '6304], 60.00th=[ 6304],\n'
                                             '     | 70.00th=[ 6368], '
                                             '80.00th=[ 6496], 90.00th=[ '
                                             '7136], 95.00th=[ 7584],\n'
                                             '     | 99.00th=[15296], '
                                             '99.50th=[18560], '
                                             '99.90th=[25984], '
                                             '99.95th=[26496],\n'
                                             '     | 99.99th=[62720]\n'
                                             '   bw (  KiB/s): min=115883, '
                                             'max=117515, per=100.00%, '
                                             'avg=116731.14, stdev=532.19, '
                                             'samples=14\n'
                                             '   iops        : min=231766, '
                                             'max=235030, avg=233462.43, '
                                             'stdev=1064.48, samples=14\n'
                                             '  lat (usec)   : 4=0.01%, '
                                             '10=97.82%, 20=1.80%, 50=0.35%, '
                                             '100=0.02%\n'
                                             '  lat (usec)   : 250=0.01%\n'
                                             '  cpu          : usr=20.03%, '
                                             'sys=57.30%, ctx=820575, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, '
                                             '2=100.0%, 4=0.0%, 8=0.0%, '
                                             '16=0.0%, 32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=1633580,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=2\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=114MiB/s (119MB/s), '
                                             '114MiB/s-114MiB/s '
                                             '(119MB/s-119MB/s), io=798MiB '
                                             '(836MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_198.output'),
                                   'state': {'begin': 1666783002.302555,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_2_libaio-xnvme_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8001463413238525,
                                             'end': 1666783003.1027014,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_198.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_198.state')},
                       'cmd_199': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_512_4_io_uring_cmd-reference_0.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_199.output'),
                                   'state': {'begin': 1666783004.1544373,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_4_io_uring_cmd-reference_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8408300876617432,
                                             'end': 1666783004.9952674,
                                             'err': 1,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_199.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_199.state')},
                       'cmd_20': {'output': 'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_20.output'),
                                  'state': {'begin': 1666782131.2885804,
                                            'cmd': '/opt/fio/bin/fio '
                                                   '--name="io_uring_cmd-bdev_xnvme" '
                                                   '--rw="randread" '
                                                   '--size="12G" --iodepth="1" '
                                                   '--bs="512" --direct="0" '
                                                   '--thread="1" '
                                                   '--time_based="1" '
                                                   '--runtime="7" '
                                                   '--ramp_time="3" '
                                                   '--norandommap="1" '
                                                   '--allow_file_create="0" '
                                                   '--ioengine="spdk_bdev" '
                                                   '--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" '
                                                   '--filename="Nvme0n1" '
                                                   '--output-format="normal,json" '
                                                   '--output="/tmp/fio-output_512_1_io_uring_cmd-bdev_xnvme_0.txt" '
                                                   '--cpus_allowed="1"',
                                            'cwd': 'None',
                                            'elapsed': 11.547215700149536,
                                            'end': 1666782142.835796,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_20.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_20.state')},
                       'cmd_200': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_200.output'),
                                   'state': {'begin': 1666783004.997414,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring_cmd-reference" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="4" --bs="512" '
                                                    '--direct="0" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="io_uring_cmd" '
                                                    '--filename="/dev/ng0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_4_io_uring_cmd-reference_0.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.062525272369385,
                                             'end': 1666783016.0599394,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_200.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_200.state')},
                       'cmd_201': {'output': 'io_uring_cmd-reference: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=io_uring_cmd, '
                                             'iodepth=4\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666790106,\n'
                                             '  "timestamp_ms" : '
                                             '1666790106140,\n'
                                             '  "time" : "Wed Oct 26 15:15:06 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring_cmd-reference",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring_cmd-reference",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "4",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "0",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"io_uring_cmd",\n'
                                             '        "filename" : '
                                             '"/dev/ng0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '1580787200,\n'
                                             '        "io_kbytes" : 1543737,\n'
                                             '        "bw_bytes" : 225794486,\n'
                                             '        "bw" : 220502,\n'
                                             '        "iops" : 441004.285102,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 3087471,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1172,\n'
                                             '          "max" : 41017,\n'
                                             '          "mean" : 1789.167935,\n'
                                             '          "stddev" : '
                                             '375.554215,\n'
                                             '          "N" : 3087472\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 100,\n'
                                             '          "max" : 122480,\n'
                                             '          "mean" : 7011.723363,\n'
                                             '          "stddev" : '
                                             '2863.847161,\n'
                                             '          "N" : 3087474,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 4128,\n'
                                             '            "5.000000" : 4704,\n'
                                             '            "10.000000" : 4768,\n'
                                             '            "20.000000" : 6112,\n'
                                             '            "30.000000" : 6304,\n'
                                             '            "40.000000" : 6368,\n'
                                             '            "50.000000" : 6432,\n'
                                             '            "60.000000" : 6624,\n'
                                             '            "70.000000" : 6944,\n'
                                             '            "80.000000" : 7584,\n'
                                             '            "90.000000" : 8512,\n'
                                             '            "95.000000" : 9280,\n'
                                             '            "99.000000" : '
                                             '20864,\n'
                                             '            "99.500000" : '
                                             '25216,\n'
                                             '            "99.900000" : '
                                             '27008,\n'
                                             '            "99.950000" : '
                                             '33536,\n'
                                             '            "99.990000" : 68096\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 5751,\n'
                                             '          "max" : 123712,\n'
                                             '          "mean" : 8800.891246,\n'
                                             '          "stddev" : '
                                             '2837.715863,\n'
                                             '          "N" : 3087474\n'
                                             '        },\n'
                                             '        "bw_min" : 219155,\n'
                                             '        "bw_max" : 221950,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '220618.285714,\n'
                                             '        "bw_dev" : 970.234264,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 438310,\n'
                                             '        "iops_max" : 443900,\n'
                                             '        "iops_mean" : '
                                             '441236.785714,\n'
                                             '        "iops_stddev" : '
                                             '1940.453523,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 15.271429,\n'
                                             '      "sys_cpu" : 80.428571,\n'
                                             '      "ctx" : 35740,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.010000,\n'
                                             '        "4" : 0.016356,\n'
                                             '        "10" : 95.917889,\n'
                                             '        "20" : 2.579652,\n'
                                             '        "50" : 1.446815,\n'
                                             '        "100" : 0.039093,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 4,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring_cmd-reference: '
                                             '(groupid=0, jobs=1): err= 0: '
                                             'pid=8364: Wed Oct 26 15:15:06 '
                                             '2022\n'
                                             '  read: IOPS=441k, BW=215MiB/s '
                                             '(226MB/s)(1508MiB/7001msec)\n'
                                             '    slat (nsec): min=1172, '
                                             'max=41017, avg=1789.17, '
                                             'stdev=375.55\n'
                                             '    clat (nsec): min=100, '
                                             'max=122480, avg=7011.72, '
                                             'stdev=2863.85\n'
                                             '     lat (usec): min=5, max=123, '
                                             'avg= 8.80, stdev= 2.84\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 4128],  '
                                             '5.00th=[ 4704], 10.00th=[ 4768], '
                                             '20.00th=[ 6112],\n'
                                             '     | 30.00th=[ 6304], '
                                             '40.00th=[ 6368], 50.00th=[ '
                                             '6432], 60.00th=[ 6624],\n'
                                             '     | 70.00th=[ 6944], '
                                             '80.00th=[ 7584], 90.00th=[ '
                                             '8512], 95.00th=[ 9280],\n'
                                             '     | 99.00th=[20864], '
                                             '99.50th=[25216], '
                                             '99.90th=[27008], '
                                             '99.95th=[33536],\n'
                                             '     | 99.99th=[68096]\n'
                                             '   bw (  KiB/s): min=219155, '
                                             'max=221950, per=100.00%, '
                                             'avg=220618.29, stdev=970.23, '
                                             'samples=14\n'
                                             '   iops        : min=438310, '
                                             'max=443900, avg=441236.79, '
                                             'stdev=1940.45, samples=14\n'
                                             '  lat (nsec)   : 250=0.01%\n'
                                             '  lat (usec)   : 2=0.01%, '
                                             '4=0.02%, 10=95.92%, 20=2.58%, '
                                             '50=1.45%\n'
                                             '  lat (usec)   : 100=0.04%, '
                                             '250=0.01%\n'
                                             '  cpu          : usr=15.27%, '
                                             'sys=80.43%, ctx=35740, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, 2=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=3087471,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=4\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=215MiB/s (226MB/s), '
                                             '215MiB/s-215MiB/s '
                                             '(226MB/s-226MB/s), io=1508MiB '
                                             '(1581MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_201.output'),
                                   'state': {'begin': 1666783016.0625746,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_4_io_uring_cmd-reference_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8646080493927002,
                                             'end': 1666783016.9271827,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_201.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_201.state')},
                       'cmd_202': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_512_4_io_uring_cmd-reference_1.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_202.output'),
                                   'state': {'begin': 1666783017.9683626,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_4_io_uring_cmd-reference_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8255369663238525,
                                             'end': 1666783018.7938995,
                                             'err': 1,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_202.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_202.state')},
                       'cmd_203': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_203.output'),
                                   'state': {'begin': 1666783018.7958016,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring_cmd-reference" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="4" --bs="512" '
                                                    '--direct="0" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="io_uring_cmd" '
                                                    '--filename="/dev/ng0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_4_io_uring_cmd-reference_1.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.042908668518066,
                                             'end': 1666783029.8387103,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_203.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_203.state')},
                       'cmd_204': {'output': 'io_uring_cmd-reference: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=io_uring_cmd, '
                                             'iodepth=4\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666790119,\n'
                                             '  "timestamp_ms" : '
                                             '1666790119919,\n'
                                             '  "time" : "Wed Oct 26 15:15:19 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring_cmd-reference",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring_cmd-reference",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "4",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "0",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"io_uring_cmd",\n'
                                             '        "filename" : '
                                             '"/dev/ng0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '1603376128,\n'
                                             '        "io_kbytes" : 1565797,\n'
                                             '        "bw_bytes" : 229021015,\n'
                                             '        "bw" : 223653,\n'
                                             '        "iops" : 447306.099129,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 3131590,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1162,\n'
                                             '          "max" : 24146,\n'
                                             '          "mean" : 1765.691817,\n'
                                             '          "stddev" : '
                                             '356.119959,\n'
                                             '          "N" : 3131591\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 90,\n'
                                             '          "max" : 125896,\n'
                                             '          "mean" : 6918.897432,\n'
                                             '          "stddev" : '
                                             '2872.697041,\n'
                                             '          "N" : 3131593,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 4080,\n'
                                             '            "5.000000" : 4640,\n'
                                             '            "10.000000" : 4704,\n'
                                             '            "20.000000" : 6048,\n'
                                             '            "30.000000" : 6176,\n'
                                             '            "40.000000" : 6240,\n'
                                             '            "50.000000" : 6304,\n'
                                             '            "60.000000" : 6432,\n'
                                             '            "70.000000" : 6816,\n'
                                             '            "80.000000" : 7456,\n'
                                             '            "90.000000" : 8512,\n'
                                             '            "95.000000" : 9152,\n'
                                             '            "99.000000" : '
                                             '20864,\n'
                                             '            "99.500000" : '
                                             '25216,\n'
                                             '            "99.900000" : '
                                             '27008,\n'
                                             '            "99.950000" : '
                                             '33024,\n'
                                             '            "99.990000" : 68096\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 5511,\n'
                                             '          "max" : 127119,\n'
                                             '          "mean" : 8684.589193,\n'
                                             '          "stddev" : '
                                             '2830.926192,\n'
                                             '          "N" : 3131593\n'
                                             '        },\n'
                                             '        "bw_min" : 222821,\n'
                                             '        "bw_max" : 224344,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '223701.928571,\n'
                                             '        "bw_dev" : 419.962354,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 445642,\n'
                                             '        "iops_max" : 448688,\n'
                                             '        "iops_mean" : '
                                             '447403.571429,\n'
                                             '        "iops_stddev" : '
                                             '839.997868,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 17.257143,\n'
                                             '      "sys_cpu" : 78.285714,\n'
                                             '      "ctx" : 39299,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.010000,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.010000,\n'
                                             '        "4" : 0.170808,\n'
                                             '        "10" : 95.608844,\n'
                                             '        "20" : 2.736278,\n'
                                             '        "50" : 1.445496,\n'
                                             '        "100" : 0.037393,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 4,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring_cmd-reference: '
                                             '(groupid=0, jobs=1): err= 0: '
                                             'pid=8410: Wed Oct 26 15:15:19 '
                                             '2022\n'
                                             '  read: IOPS=447k, BW=218MiB/s '
                                             '(229MB/s)(1529MiB/7001msec)\n'
                                             '    slat (nsec): min=1162, '
                                             'max=24146, avg=1765.69, '
                                             'stdev=356.12\n'
                                             '    clat (nsec): min=90, '
                                             'max=125896, avg=6918.90, '
                                             'stdev=2872.70\n'
                                             '     lat (usec): min=5, max=127, '
                                             'avg= 8.68, stdev= 2.83\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 4080],  '
                                             '5.00th=[ 4640], 10.00th=[ 4704], '
                                             '20.00th=[ 6048],\n'
                                             '     | 30.00th=[ 6176], '
                                             '40.00th=[ 6240], 50.00th=[ '
                                             '6304], 60.00th=[ 6432],\n'
                                             '     | 70.00th=[ 6816], '
                                             '80.00th=[ 7456], 90.00th=[ '
                                             '8512], 95.00th=[ 9152],\n'
                                             '     | 99.00th=[20864], '
                                             '99.50th=[25216], '
                                             '99.90th=[27008], '
                                             '99.95th=[33024],\n'
                                             '     | 99.99th=[68096]\n'
                                             '   bw (  KiB/s): min=222821, '
                                             'max=224344, per=100.00%, '
                                             'avg=223701.93, stdev=419.96, '
                                             'samples=14\n'
                                             '   iops        : min=445642, '
                                             'max=448688, avg=447403.57, '
                                             'stdev=840.00, samples=14\n'
                                             '  lat (nsec)   : 100=0.01%, '
                                             '250=0.01%\n'
                                             '  lat (usec)   : 2=0.01%, '
                                             '4=0.17%, 10=95.61%, 20=2.74%, '
                                             '50=1.45%\n'
                                             '  lat (usec)   : 100=0.04%, '
                                             '250=0.01%\n'
                                             '  cpu          : usr=17.26%, '
                                             'sys=78.29%, ctx=39299, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, 2=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=3131590,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=4\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=218MiB/s (229MB/s), '
                                             '218MiB/s-218MiB/s '
                                             '(229MB/s-229MB/s), io=1529MiB '
                                             '(1603MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_204.output'),
                                   'state': {'begin': 1666783029.841205,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_4_io_uring_cmd-reference_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8257076740264893,
                                             'end': 1666783030.6669126,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_204.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_204.state')},
                       'cmd_205': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_512_4_io_uring_cmd-reference_2.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_205.output'),
                                   'state': {'begin': 1666783031.7331214,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_4_io_uring_cmd-reference_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8146262168884277,
                                             'end': 1666783032.5477476,
                                             'err': 1,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_205.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_205.state')},
                       'cmd_206': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_206.output'),
                                   'state': {'begin': 1666783032.5496573,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring_cmd-reference" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="4" --bs="512" '
                                                    '--direct="0" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="io_uring_cmd" '
                                                    '--filename="/dev/ng0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_4_io_uring_cmd-reference_2.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.072738885879517,
                                             'end': 1666783043.6223962,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_206.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_206.state')},
                       'cmd_207': {'output': 'io_uring_cmd-reference: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=io_uring_cmd, '
                                             'iodepth=4\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666790133,\n'
                                             '  "timestamp_ms" : '
                                             '1666790133702,\n'
                                             '  "time" : "Wed Oct 26 15:15:33 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring_cmd-reference",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring_cmd-reference",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "4",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "0",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"io_uring_cmd",\n'
                                             '        "filename" : '
                                             '"/dev/ng0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '1593121280,\n'
                                             '        "io_kbytes" : 1555782,\n'
                                             '        "bw_bytes" : 227556246,\n'
                                             '        "bw" : 222222,\n'
                                             '        "iops" : 444445.222111,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 3111561,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1162,\n'
                                             '          "max" : 18786,\n'
                                             '          "mean" : 1779.421280,\n'
                                             '          "stddev" : '
                                             '358.143700,\n'
                                             '          "N" : 3111562\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 90,\n'
                                             '          "max" : 126347,\n'
                                             '          "mean" : 6962.273469,\n'
                                             '          "stddev" : '
                                             '2892.549055,\n'
                                             '          "N" : 3111564,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 4128,\n'
                                             '            "5.000000" : 4640,\n'
                                             '            "10.000000" : 4768,\n'
                                             '            "20.000000" : 6112,\n'
                                             '            "30.000000" : 6176,\n'
                                             '            "40.000000" : 6304,\n'
                                             '            "50.000000" : 6368,\n'
                                             '            "60.000000" : 6496,\n'
                                             '            "70.000000" : 6880,\n'
                                             '            "80.000000" : 7456,\n'
                                             '            "90.000000" : 8512,\n'
                                             '            "95.000000" : 9152,\n'
                                             '            "99.000000" : '
                                             '20864,\n'
                                             '            "99.500000" : '
                                             '25472,\n'
                                             '            "99.900000" : '
                                             '27264,\n'
                                             '            "99.950000" : '
                                             '34048,\n'
                                             '            "99.990000" : 68096\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 5530,\n'
                                             '          "max" : 127549,\n'
                                             '          "mean" : 8741.694742,\n'
                                             '          "stddev" : '
                                             '2854.750029,\n'
                                             '          "N" : 3111564\n'
                                             '        },\n'
                                             '        "bw_min" : 221009,\n'
                                             '        "bw_max" : 223129,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '222349.285714,\n'
                                             '        "bw_dev" : 743.621536,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 442018,\n'
                                             '        "iops_max" : 446258,\n'
                                             '        "iops_mean" : '
                                             '444698.428571,\n'
                                             '        "iops_stddev" : '
                                             '1487.477820,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 16.157143,\n'
                                             '      "sys_cpu" : 79.471429,\n'
                                             '      "ctx" : 36179,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.010000,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.010000,\n'
                                             '        "4" : 0.077582,\n'
                                             '        "10" : 95.868408,\n'
                                             '        "20" : 2.476474,\n'
                                             '        "50" : 1.537267,\n'
                                             '        "100" : 0.039498,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 4,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring_cmd-reference: '
                                             '(groupid=0, jobs=1): err= 0: '
                                             'pid=8456: Wed Oct 26 15:15:33 '
                                             '2022\n'
                                             '  read: IOPS=444k, BW=217MiB/s '
                                             '(228MB/s)(1519MiB/7001msec)\n'
                                             '    slat (nsec): min=1162, '
                                             'max=18786, avg=1779.42, '
                                             'stdev=358.14\n'
                                             '    clat (nsec): min=90, '
                                             'max=126347, avg=6962.27, '
                                             'stdev=2892.55\n'
                                             '     lat (usec): min=5, max=127, '
                                             'avg= 8.74, stdev= 2.85\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 4128],  '
                                             '5.00th=[ 4640], 10.00th=[ 4768], '
                                             '20.00th=[ 6112],\n'
                                             '     | 30.00th=[ 6176], '
                                             '40.00th=[ 6304], 50.00th=[ '
                                             '6368], 60.00th=[ 6496],\n'
                                             '     | 70.00th=[ 6880], '
                                             '80.00th=[ 7456], 90.00th=[ '
                                             '8512], 95.00th=[ 9152],\n'
                                             '     | 99.00th=[20864], '
                                             '99.50th=[25472], '
                                             '99.90th=[27264], '
                                             '99.95th=[34048],\n'
                                             '     | 99.99th=[68096]\n'
                                             '   bw (  KiB/s): min=221009, '
                                             'max=223129, per=100.00%, '
                                             'avg=222349.29, stdev=743.62, '
                                             'samples=14\n'
                                             '   iops        : min=442018, '
                                             'max=446258, avg=444698.43, '
                                             'stdev=1487.48, samples=14\n'
                                             '  lat (nsec)   : 100=0.01%, '
                                             '250=0.01%\n'
                                             '  lat (usec)   : 2=0.01%, '
                                             '4=0.08%, 10=95.87%, 20=2.48%, '
                                             '50=1.54%\n'
                                             '  lat (usec)   : 100=0.04%, '
                                             '250=0.01%\n'
                                             '  cpu          : usr=16.16%, '
                                             'sys=79.47%, ctx=36179, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, 2=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=3111561,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=4\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=217MiB/s (228MB/s), '
                                             '217MiB/s-217MiB/s '
                                             '(228MB/s-228MB/s), io=1519MiB '
                                             '(1593MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_207.output'),
                                   'state': {'begin': 1666783043.6253064,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_4_io_uring_cmd-reference_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7937602996826172,
                                             'end': 1666783044.4190667,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_207.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_207.state')},
                       'cmd_208': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_512_4_io_uring_cmd-xnvme_0.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_208.output'),
                                   'state': {'begin': 1666783045.4686837,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_4_io_uring_cmd-xnvme_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7892277240753174,
                                             'end': 1666783046.2579114,
                                             'err': 1,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_208.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_208.state')},
                       'cmd_209': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_209.output'),
                                   'state': {'begin': 1666783046.2600665,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring_cmd-xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="4" --bs="512" '
                                                    '--direct="0" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
                                                    '--xnvme_be="linux" '
                                                    '--xnvme_async="io_uring_cmd" '
                                                    '--xnvme_sync="nvme" '
                                                    '--xnvme_admin="nvme" '
                                                    '--xnvme_dev_nsid="1" '
                                                    '--filename="/dev/ng0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_4_io_uring_cmd-xnvme_0.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.105571985244751,
                                             'end': 1666783057.3656385,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_209.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_209.state')},
                       'cmd_21': {'output': 'io_uring_cmd-bdev_xnvme: (g=0): '
                                            'rw=randread, bs=(R) 512B-512B, '
                                            '(W) 512B-512B, (T) 512B-512B, '
                                            'ioengine=spdk_bdev, iodepth=1\n'
                                            'fio-3.32-dirty\n'
                                            'Starting 1 thread\n'
                                            '{\n'
                                            '  "fio version" : '
                                            '"fio-3.32-dirty",\n'
                                            '  "timestamp" : 1666789232,\n'
                                            '  "timestamp_ms" : '
                                            '1666789232866,\n'
                                            '  "time" : "Wed Oct 26 15:00:32 '
                                            '2022",\n'
                                            '  "jobs" : [\n'
                                            '    {\n'
                                            '      "jobname" : '
                                            '"io_uring_cmd-bdev_xnvme",\n'
                                            '      "groupid" : 0,\n'
                                            '      "error" : 0,\n'
                                            '      "eta" : 0,\n'
                                            '      "elapsed" : 11,\n'
                                            '      "job options" : {\n'
                                            '        "name" : '
                                            '"io_uring_cmd-bdev_xnvme",\n'
                                            '        "rw" : "randread",\n'
                                            '        "size" : "12G",\n'
                                            '        "iodepth" : "1",\n'
                                            '        "bs" : "512",\n'
                                            '        "direct" : "0",\n'
                                            '        "thread" : "1",\n'
                                            '        "time_based" : "1",\n'
                                            '        "runtime" : "7",\n'
                                            '        "ramp_time" : "3",\n'
                                            '        "norandommap" : "1",\n'
                                            '        "allow_file_create" : '
                                            '"0",\n'
                                            '        "ioengine" : '
                                            '"spdk_bdev",\n'
                                            '        "spdk_conf" : '
                                            '"/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",\n'
                                            '        "filename" : "Nvme0n1",\n'
                                            '        "cpus_allowed" : "1"\n'
                                            '      },\n'
                                            '      "read" : {\n'
                                            '        "io_bytes" : 584781824,\n'
                                            '        "io_kbytes" : 571076,\n'
                                            '        "bw_bytes" : 83528327,\n'
                                            '        "bw" : 81570,\n'
                                            '        "iops" : 163141.265533,\n'
                                            '        "runtime" : 7001,\n'
                                            '        "total_ios" : 1142152,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 230,\n'
                                            '          "max" : 3617,\n'
                                            '          "mean" : 269.127672,\n'
                                            '          "stddev" : 21.625019,\n'
                                            '          "N" : 1142152\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 2775,\n'
                                            '          "max" : 111990,\n'
                                            '          "mean" : 5644.366970,\n'
                                            '          "stddev" : '
                                            '1128.740165,\n'
                                            '          "N" : 1142152,\n'
                                            '          "percentile" : {\n'
                                            '            "1.000000" : 5344,\n'
                                            '            "5.000000" : 5344,\n'
                                            '            "10.000000" : 5408,\n'
                                            '            "20.000000" : 5408,\n'
                                            '            "30.000000" : 5472,\n'
                                            '            "40.000000" : 5472,\n'
                                            '            "50.000000" : 5472,\n'
                                            '            "60.000000" : 5536,\n'
                                            '            "70.000000" : 5536,\n'
                                            '            "80.000000" : 5600,\n'
                                            '            "90.000000" : 5664,\n'
                                            '            "95.000000" : 5728,\n'
                                            '            "99.000000" : 11456,\n'
                                            '            "99.500000" : 12480,\n'
                                            '            "99.900000" : 16768,\n'
                                            '            "99.950000" : 17024,\n'
                                            '            "99.990000" : 58624\n'
                                            '          }\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 5560,\n'
                                            '          "max" : 112270,\n'
                                            '          "mean" : 5913.494643,\n'
                                            '          "stddev" : '
                                            '1129.505716,\n'
                                            '          "N" : 1142152\n'
                                            '        },\n'
                                            '        "bw_min" : 81452,\n'
                                            '        "bw_max" : 81832,\n'
                                            '        "bw_agg" : 100.000000,\n'
                                            '        "bw_mean" : '
                                            '81605.142857,\n'
                                            '        "bw_dev" : 116.354139,\n'
                                            '        "bw_samples" : 14,\n'
                                            '        "iops_min" : 162905,\n'
                                            '        "iops_max" : 163665,\n'
                                            '        "iops_mean" : '
                                            '163210.428571,\n'
                                            '        "iops_stddev" : '
                                            '232.757290,\n'
                                            '        "iops_samples" : 14\n'
                                            '      },\n'
                                            '      "write" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "trim" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "sync" : {\n'
                                            '        "total_ios" : 0,\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        }\n'
                                            '      },\n'
                                            '      "job_runtime" : 7000,\n'
                                            '      "usr_cpu" : 99.985714,\n'
                                            '      "sys_cpu" : 0.000000,\n'
                                            '      "ctx" : 13,\n'
                                            '      "majf" : 0,\n'
                                            '      "minf" : 0,\n'
                                            '      "iodepth_level" : {\n'
                                            '        "1" : 100.000000,\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_submit" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_complete" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ns" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_us" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.010000,\n'
                                            '        "10" : 98.485491,\n'
                                            '        "20" : 1.498575,\n'
                                            '        "50" : 0.010000,\n'
                                            '        "100" : 0.013221,\n'
                                            '        "250" : 0.010000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ms" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000,\n'
                                            '        "2000" : 0.000000,\n'
                                            '        ">=2000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_depth" : 1,\n'
                                            '      "latency_target" : 0,\n'
                                            '      "latency_percentile" : '
                                            '100.000000,\n'
                                            '      "latency_window" : 0\n'
                                            '    }\n'
                                            '  ]\n'
                                            '}\n'
                                            '\n'
                                            'io_uring_cmd-bdev_xnvme: '
                                            '(groupid=0, jobs=1): err= 0: '
                                            'pid=5227: Wed Oct 26 15:00:32 '
                                            '2022\n'
                                            '  read: IOPS=163k, BW=79.7MiB/s '
                                            '(83.5MB/s)(558MiB/7001msec)\n'
                                            '    slat (nsec): min=230, '
                                            'max=3617, avg=269.13, '
                                            'stdev=21.63\n'
                                            '    clat (usec): min=2, max=111, '
                                            'avg= 5.64, stdev= 1.13\n'
                                            '     lat (usec): min=5, max=112, '
                                            'avg= 5.91, stdev= 1.13\n'
                                            '    clat percentiles (nsec):\n'
                                            '     |  1.00th=[ 5344],  5.00th=[ '
                                            '5344], 10.00th=[ 5408], 20.00th=[ '
                                            '5408],\n'
                                            '     | 30.00th=[ 5472], 40.00th=[ '
                                            '5472], 50.00th=[ 5472], 60.00th=[ '
                                            '5536],\n'
                                            '     | 70.00th=[ 5536], 80.00th=[ '
                                            '5600], 90.00th=[ 5664], 95.00th=[ '
                                            '5728],\n'
                                            '     | 99.00th=[11456], '
                                            '99.50th=[12480], 99.90th=[16768], '
                                            '99.95th=[17024],\n'
                                            '     | 99.99th=[58624]\n'
                                            '   bw (  KiB/s): min=81452, '
                                            'max=81832, per=100.00%, '
                                            'avg=81605.14, stdev=116.35, '
                                            'samples=14\n'
                                            '   iops        : min=162905, '
                                            'max=163665, avg=163210.43, '
                                            'stdev=232.76, samples=14\n'
                                            '  lat (usec)   : 4=0.01%, '
                                            '10=98.49%, 20=1.50%, 50=0.01%, '
                                            '100=0.01%\n'
                                            '  lat (usec)   : 250=0.01%\n'
                                            '  cpu          : usr=99.99%, '
                                            'sys=0.00%, ctx=13, majf=0, '
                                            'minf=0\n'
                                            '  IO depths    : 1=100.0%, '
                                            '2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, >=64=0.0%\n'
                                            '     submit    : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     complete  : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     issued rwts: '
                                            'total=1142152,0,0,0 short=0,0,0,0 '
                                            'dropped=0,0,0,0\n'
                                            '     latency   : target=0, '
                                            'window=0, percentile=100.00%, '
                                            'depth=1\n'
                                            '\n'
                                            'Run status group 0 (all jobs):\n'
                                            '   READ: bw=79.7MiB/s (83.5MB/s), '
                                            '79.7MiB/s-79.7MiB/s '
                                            '(83.5MB/s-83.5MB/s), io=558MiB '
                                            '(585MB), run=7001-7001msec\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_21.output'),
                                  'state': {'begin': 1666782142.8388188,
                                            'cmd': 'cat '
                                                   '/tmp/fio-output_512_1_io_uring_cmd-bdev_xnvme_0.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.8268139362335205,
                                            'end': 1666782143.6656327,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_21.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_21.state')},
                       'cmd_210': {'output': 'io_uring_cmd-xnvme: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=xnvme, iodepth=4\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666790147,\n'
                                             '  "timestamp_ms" : '
                                             '1666790147445,\n'
                                             '  "time" : "Wed Oct 26 15:15:47 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring_cmd-xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring_cmd-xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "4",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "0",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
                                             '        "xnvme_be" : "linux",\n'
                                             '        "xnvme_async" : '
                                             '"io_uring_cmd",\n'
                                             '        "xnvme_sync" : "nvme",\n'
                                             '        "xnvme_admin" : "nvme",\n'
                                             '        "xnvme_dev_nsid" : "1",\n'
                                             '        "filename" : '
                                             '"/dev/ng0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '1479533568,\n'
                                             '        "io_kbytes" : 1444857,\n'
                                             '        "bw_bytes" : 211331748,\n'
                                             '        "bw" : 206378,\n'
                                             '        "iops" : 412756.891873,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 2889711,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1232,\n'
                                             '          "max" : 22802,\n'
                                             '          "mean" : 1856.760263,\n'
                                             '          "stddev" : '
                                             '376.862601,\n'
                                             '          "N" : 2889711\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 140,\n'
                                             '          "max" : 125245,\n'
                                             '          "mean" : 7527.013691,\n'
                                             '          "stddev" : '
                                             '2833.703087,\n'
                                             '          "N" : 2889714,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 4320,\n'
                                             '            "5.000000" : 4896,\n'
                                             '            "10.000000" : 5792,\n'
                                             '            "20.000000" : 6624,\n'
                                             '            "30.000000" : 6688,\n'
                                             '            "40.000000" : 7136,\n'
                                             '            "50.000000" : 7264,\n'
                                             '            "60.000000" : 7328,\n'
                                             '            "70.000000" : 7520,\n'
                                             '            "80.000000" : 7904,\n'
                                             '            "90.000000" : 8512,\n'
                                             '            "95.000000" : 9408,\n'
                                             '            "99.000000" : '
                                             '20864,\n'
                                             '            "99.500000" : '
                                             '25216,\n'
                                             '            "99.900000" : '
                                             '27520,\n'
                                             '            "99.950000" : '
                                             '34048,\n'
                                             '            "99.990000" : 69120\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 5701,\n'
                                             '          "max" : 127048,\n'
                                             '          "mean" : 9383.773646,\n'
                                             '          "stddev" : '
                                             '2814.443814,\n'
                                             '          "N" : 2889714\n'
                                             '        },\n'
                                             '        "bw_min" : 205006,\n'
                                             '        "bw_max" : 208277,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '206571.076923,\n'
                                             '        "bw_dev" : 1108.539088,\n'
                                             '        "bw_samples" : 13,\n'
                                             '        "iops_min" : 410012,\n'
                                             '        "iops_max" : 416554,\n'
                                             '        "iops_mean" : '
                                             '413142.153846,\n'
                                             '        "iops_stddev" : '
                                             '2216.807195,\n'
                                             '        "iops_samples" : 13\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 19.342857,\n'
                                             '      "sys_cpu" : 76.642857,\n'
                                             '      "ctx" : 35152,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 95.738432,\n'
                                             '        "20" : 2.739409,\n'
                                             '        "50" : 1.478556,\n'
                                             '        "100" : 0.041319,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 4,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring_cmd-xnvme: (groupid=0, '
                                             'jobs=1): err= 0: pid=8503: Wed '
                                             'Oct 26 15:15:47 2022\n'
                                             '  read: IOPS=413k, BW=202MiB/s '
                                             '(211MB/s)(1411MiB/7001msec)\n'
                                             '    slat (nsec): min=1232, '
                                             'max=22802, avg=1856.76, '
                                             'stdev=376.86\n'
                                             '    clat (nsec): min=140, '
                                             'max=125245, avg=7527.01, '
                                             'stdev=2833.70\n'
                                             '     lat (usec): min=5, max=127, '
                                             'avg= 9.38, stdev= 2.81\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 4320],  '
                                             '5.00th=[ 4896], 10.00th=[ 5792], '
                                             '20.00th=[ 6624],\n'
                                             '     | 30.00th=[ 6688], '
                                             '40.00th=[ 7136], 50.00th=[ '
                                             '7264], 60.00th=[ 7328],\n'
                                             '     | 70.00th=[ 7520], '
                                             '80.00th=[ 7904], 90.00th=[ '
                                             '8512], 95.00th=[ 9408],\n'
                                             '     | 99.00th=[20864], '
                                             '99.50th=[25216], '
                                             '99.90th=[27520], '
                                             '99.95th=[34048],\n'
                                             '     | 99.99th=[69120]\n'
                                             '   bw (  KiB/s): min=205006, '
                                             'max=208277, per=100.00%, '
                                             'avg=206571.08, stdev=1108.54, '
                                             'samples=13\n'
                                             '   iops        : min=410012, '
                                             'max=416554, avg=413142.15, '
                                             'stdev=2216.81, samples=13\n'
                                             '  lat (nsec)   : 250=0.01%\n'
                                             '  lat (usec)   : 4=0.01%, '
                                             '10=95.74%, 20=2.74%, 50=1.48%, '
                                             '100=0.04%\n'
                                             '  lat (usec)   : 250=0.01%\n'
                                             '  cpu          : usr=19.34%, '
                                             'sys=76.64%, ctx=35152, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, 2=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=2889711,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=4\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=202MiB/s (211MB/s), '
                                             '202MiB/s-202MiB/s '
                                             '(211MB/s-211MB/s), io=1411MiB '
                                             '(1480MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_210.output'),
                                   'state': {'begin': 1666783057.3694043,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_4_io_uring_cmd-xnvme_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.842477560043335,
                                             'end': 1666783058.2118819,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_210.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_210.state')},
                       'cmd_211': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_512_4_io_uring_cmd-xnvme_1.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_211.output'),
                                   'state': {'begin': 1666783059.212763,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_4_io_uring_cmd-xnvme_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7731776237487793,
                                             'end': 1666783059.9859407,
                                             'err': 1,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_211.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_211.state')},
                       'cmd_212': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_212.output'),
                                   'state': {'begin': 1666783059.9877508,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring_cmd-xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="4" --bs="512" '
                                                    '--direct="0" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
                                                    '--xnvme_be="linux" '
                                                    '--xnvme_async="io_uring_cmd" '
                                                    '--xnvme_sync="nvme" '
                                                    '--xnvme_admin="nvme" '
                                                    '--xnvme_dev_nsid="1" '
                                                    '--filename="/dev/ng0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_4_io_uring_cmd-xnvme_1.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.118003368377686,
                                             'end': 1666783071.1057541,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_212.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_212.state')},
                       'cmd_213': {'output': 'io_uring_cmd-xnvme: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=xnvme, iodepth=4\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666790161,\n'
                                             '  "timestamp_ms" : '
                                             '1666790161185,\n'
                                             '  "time" : "Wed Oct 26 15:16:01 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring_cmd-xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring_cmd-xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "4",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "0",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
                                             '        "xnvme_be" : "linux",\n'
                                             '        "xnvme_async" : '
                                             '"io_uring_cmd",\n'
                                             '        "xnvme_sync" : "nvme",\n'
                                             '        "xnvme_admin" : "nvme",\n'
                                             '        "xnvme_dev_nsid" : "1",\n'
                                             '        "filename" : '
                                             '"/dev/ng0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '1513281024,\n'
                                             '        "io_kbytes" : 1477813,\n'
                                             '        "bw_bytes" : 216152124,\n'
                                             '        "bw" : 211086,\n'
                                             '        "iops" : 422171.546922,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 2955623,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1212,\n'
                                             '          "max" : 68398,\n'
                                             '          "mean" : 1804.931661,\n'
                                             '          "stddev" : '
                                             '383.353388,\n'
                                             '          "N" : 2955624\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 130,\n'
                                             '          "max" : 121648,\n'
                                             '          "mean" : 7361.880692,\n'
                                             '          "stddev" : '
                                             '2801.249674,\n'
                                             '          "N" : 2955626,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 4256,\n'
                                             '            "5.000000" : 4832,\n'
                                             '            "10.000000" : 5088,\n'
                                             '            "20.000000" : 6496,\n'
                                             '            "30.000000" : 6560,\n'
                                             '            "40.000000" : 6688,\n'
                                             '            "50.000000" : 7072,\n'
                                             '            "60.000000" : 7200,\n'
                                             '            "70.000000" : 7392,\n'
                                             '            "80.000000" : 7840,\n'
                                             '            "90.000000" : 8384,\n'
                                             '            "95.000000" : 9280,\n'
                                             '            "99.000000" : '
                                             '20864,\n'
                                             '            "99.500000" : '
                                             '24960,\n'
                                             '            "99.900000" : '
                                             '27264,\n'
                                             '            "99.950000" : '
                                             '33024,\n'
                                             '            "99.990000" : 68096\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 5770,\n'
                                             '          "max" : 122910,\n'
                                             '          "mean" : 9166.812258,\n'
                                             '          "stddev" : '
                                             '2786.079322,\n'
                                             '          "N" : 2955626\n'
                                             '        },\n'
                                             '        "bw_min" : 209124,\n'
                                             '        "bw_max" : 212490,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '211123.571429,\n'
                                             '        "bw_dev" : 1103.367274,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 418248,\n'
                                             '        "iops_max" : 424980,\n'
                                             '        "iops_mean" : '
                                             '422247.142857,\n'
                                             '        "iops_stddev" : '
                                             '2206.773449,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 19.828571,\n'
                                             '      "sys_cpu" : 76.085714,\n'
                                             '      "ctx" : 35312,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 1,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.010000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 95.890816,\n'
                                             '        "20" : 2.646379,\n'
                                             '        "50" : 1.422881,\n'
                                             '        "100" : 0.037623,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 4,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring_cmd-xnvme: (groupid=0, '
                                             'jobs=1): err= 0: pid=8549: Wed '
                                             'Oct 26 15:16:01 2022\n'
                                             '  read: IOPS=422k, BW=206MiB/s '
                                             '(216MB/s)(1443MiB/7001msec)\n'
                                             '    slat (nsec): min=1212, '
                                             'max=68398, avg=1804.93, '
                                             'stdev=383.35\n'
                                             '    clat (nsec): min=130, '
                                             'max=121648, avg=7361.88, '
                                             'stdev=2801.25\n'
                                             '     lat (usec): min=5, max=122, '
                                             'avg= 9.17, stdev= 2.79\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 4256],  '
                                             '5.00th=[ 4832], 10.00th=[ 5088], '
                                             '20.00th=[ 6496],\n'
                                             '     | 30.00th=[ 6560], '
                                             '40.00th=[ 6688], 50.00th=[ '
                                             '7072], 60.00th=[ 7200],\n'
                                             '     | 70.00th=[ 7392], '
                                             '80.00th=[ 7840], 90.00th=[ '
                                             '8384], 95.00th=[ 9280],\n'
                                             '     | 99.00th=[20864], '
                                             '99.50th=[24960], '
                                             '99.90th=[27264], '
                                             '99.95th=[33024],\n'
                                             '     | 99.99th=[68096]\n'
                                             '   bw (  KiB/s): min=209124, '
                                             'max=212490, per=100.00%, '
                                             'avg=211123.57, stdev=1103.37, '
                                             'samples=14\n'
                                             '   iops        : min=418248, '
                                             'max=424980, avg=422247.14, '
                                             'stdev=2206.77, samples=14\n'
                                             '  lat (nsec)   : 250=0.01%\n'
                                             '  lat (usec)   : 2=0.01%, '
                                             '4=0.01%, 10=95.89%, 20=2.65%, '
                                             '50=1.42%\n'
                                             '  lat (usec)   : 100=0.04%, '
                                             '250=0.01%\n'
                                             '  cpu          : usr=19.83%, '
                                             'sys=76.09%, ctx=35312, majf=0, '
                                             'minf=1\n'
                                             '  IO depths    : 1=0.0%, 2=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=2955623,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=4\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=206MiB/s (216MB/s), '
                                             '206MiB/s-206MiB/s '
                                             '(216MB/s-216MB/s), io=1443MiB '
                                             '(1513MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_213.output'),
                                   'state': {'begin': 1666783071.1096673,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_4_io_uring_cmd-xnvme_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.9034547805786133,
                                             'end': 1666783072.013122,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_213.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_213.state')},
                       'cmd_214': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_512_4_io_uring_cmd-xnvme_2.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_214.output'),
                                   'state': {'begin': 1666783073.0147161,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_4_io_uring_cmd-xnvme_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7947611808776855,
                                             'end': 1666783073.8094773,
                                             'err': 1,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_214.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_214.state')},
                       'cmd_215': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_215.output'),
                                   'state': {'begin': 1666783073.8116484,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring_cmd-xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="4" --bs="512" '
                                                    '--direct="0" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
                                                    '--xnvme_be="linux" '
                                                    '--xnvme_async="io_uring_cmd" '
                                                    '--xnvme_sync="nvme" '
                                                    '--xnvme_admin="nvme" '
                                                    '--xnvme_dev_nsid="1" '
                                                    '--filename="/dev/ng0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_4_io_uring_cmd-xnvme_2.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.099498748779297,
                                             'end': 1666783084.911147,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_215.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_215.state')},
                       'cmd_216': {'output': 'io_uring_cmd-xnvme: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=xnvme, iodepth=4\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666790174,\n'
                                             '  "timestamp_ms" : '
                                             '1666790174990,\n'
                                             '  "time" : "Wed Oct 26 15:16:14 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring_cmd-xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring_cmd-xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "4",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "0",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
                                             '        "xnvme_be" : "linux",\n'
                                             '        "xnvme_async" : '
                                             '"io_uring_cmd",\n'
                                             '        "xnvme_sync" : "nvme",\n'
                                             '        "xnvme_admin" : "nvme",\n'
                                             '        "xnvme_dev_nsid" : "1",\n'
                                             '        "filename" : '
                                             '"/dev/ng0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '1493146112,\n'
                                             '        "io_kbytes" : 1458150,\n'
                                             '        "bw_bytes" : 213276119,\n'
                                             '        "bw" : 208277,\n'
                                             '        "iops" : 416554.349379,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 2916297,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1222,\n'
                                             '          "max" : 23835,\n'
                                             '          "mean" : 1835.246559,\n'
                                             '          "stddev" : '
                                             '380.983468,\n'
                                             '          "N" : 2916298\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 140,\n'
                                             '          "max" : 117631,\n'
                                             '          "mean" : 7459.801219,\n'
                                             '          "stddev" : '
                                             '2783.673537,\n'
                                             '          "N" : 2916300,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 4256,\n'
                                             '            "5.000000" : 4896,\n'
                                             '            "10.000000" : 5792,\n'
                                             '            "20.000000" : 6560,\n'
                                             '            "30.000000" : 6624,\n'
                                             '            "40.000000" : 6816,\n'
                                             '            "50.000000" : 7200,\n'
                                             '            "60.000000" : 7264,\n'
                                             '            "70.000000" : 7520,\n'
                                             '            "80.000000" : 7904,\n'
                                             '            "90.000000" : 8384,\n'
                                             '            "95.000000" : 9408,\n'
                                             '            "99.000000" : '
                                             '20864,\n'
                                             '            "99.500000" : '
                                             '25216,\n'
                                             '            "99.900000" : '
                                             '27264,\n'
                                             '            "99.950000" : '
                                             '33024,\n'
                                             '            "99.990000" : 68096\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 5740,\n'
                                             '          "max" : 119665,\n'
                                             '          "mean" : 9295.047879,\n'
                                             '          "stddev" : '
                                             '2765.898562,\n'
                                             '          "N" : 2916300\n'
                                             '        },\n'
                                             '        "bw_min" : 206224,\n'
                                             '        "bw_max" : 211144,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '208366.357143,\n'
                                             '        "bw_dev" : 1490.058678,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 412448,\n'
                                             '        "iops_max" : 422288,\n'
                                             '        "iops_mean" : '
                                             '416732.642857,\n'
                                             '        "iops_stddev" : '
                                             '2980.303175,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 20.971429,\n'
                                             '      "sys_cpu" : 75.014286,\n'
                                             '      "ctx" : 35185,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 95.807011,\n'
                                             '        "20" : 2.722768,\n'
                                             '        "50" : 1.432056,\n'
                                             '        "100" : 0.036450,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 4,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring_cmd-xnvme: (groupid=0, '
                                             'jobs=1): err= 0: pid=8596: Wed '
                                             'Oct 26 15:16:14 2022\n'
                                             '  read: IOPS=417k, BW=203MiB/s '
                                             '(213MB/s)(1424MiB/7001msec)\n'
                                             '    slat (nsec): min=1222, '
                                             'max=23835, avg=1835.25, '
                                             'stdev=380.98\n'
                                             '    clat (nsec): min=140, '
                                             'max=117631, avg=7459.80, '
                                             'stdev=2783.67\n'
                                             '     lat (usec): min=5, max=119, '
                                             'avg= 9.30, stdev= 2.77\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 4256],  '
                                             '5.00th=[ 4896], 10.00th=[ 5792], '
                                             '20.00th=[ 6560],\n'
                                             '     | 30.00th=[ 6624], '
                                             '40.00th=[ 6816], 50.00th=[ '
                                             '7200], 60.00th=[ 7264],\n'
                                             '     | 70.00th=[ 7520], '
                                             '80.00th=[ 7904], 90.00th=[ '
                                             '8384], 95.00th=[ 9408],\n'
                                             '     | 99.00th=[20864], '
                                             '99.50th=[25216], '
                                             '99.90th=[27264], '
                                             '99.95th=[33024],\n'
                                             '     | 99.99th=[68096]\n'
                                             '   bw (  KiB/s): min=206224, '
                                             'max=211144, per=100.00%, '
                                             'avg=208366.36, stdev=1490.06, '
                                             'samples=14\n'
                                             '   iops        : min=412448, '
                                             'max=422288, avg=416732.64, '
                                             'stdev=2980.30, samples=14\n'
                                             '  lat (nsec)   : 250=0.01%\n'
                                             '  lat (usec)   : 4=0.01%, '
                                             '10=95.81%, 20=2.72%, 50=1.43%, '
                                             '100=0.04%\n'
                                             '  lat (usec)   : 250=0.01%\n'
                                             '  cpu          : usr=20.97%, '
                                             'sys=75.01%, ctx=35185, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, 2=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=2916297,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=4\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=203MiB/s (213MB/s), '
                                             '203MiB/s-203MiB/s '
                                             '(213MB/s-213MB/s), io=1424MiB '
                                             '(1493MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_216.output'),
                                   'state': {'begin': 1666783084.9171014,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_4_io_uring_cmd-xnvme_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8301782608032227,
                                             'end': 1666783085.7472796,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_216.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_216.state')},
                       'cmd_217': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_512_4_io_uring_cmd-bdev_xnvme_0.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_217.output'),
                                   'state': {'begin': 1666783087.8358128,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_4_io_uring_cmd-bdev_xnvme_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8178656101226807,
                                             'end': 1666783088.6536784,
                                             'err': 1,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_217.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_217.state')},
                       'cmd_218': {'output': 'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_218.output'),
                                   'state': {'begin': 1666783088.6558793,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring_cmd-bdev_xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="4" --bs="512" '
                                                    '--direct="0" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="spdk_bdev" '
                                                    '--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" '
                                                    '--filename="Nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_4_io_uring_cmd-bdev_xnvme_0.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.464780807495117,
                                             'end': 1666783100.12066,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_218.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_218.state')},
                       'cmd_219': {'output': 'io_uring_cmd-bdev_xnvme: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=spdk_bdev, iodepth=4\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666790190,\n'
                                             '  "timestamp_ms" : '
                                             '1666790190135,\n'
                                             '  "time" : "Wed Oct 26 15:16:30 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring_cmd-bdev_xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring_cmd-bdev_xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "4",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "0",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"spdk_bdev",\n'
                                             '        "spdk_conf" : '
                                             '"/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",\n'
                                             '        "filename" : "Nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '1345384448,\n'
                                             '        "io_kbytes" : 1313852,\n'
                                             '        "bw_bytes" : 192170325,\n'
                                             '        "bw" : 187666,\n'
                                             '        "iops" : 375332.238252,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 2627701,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 651,\n'
                                             '          "max" : 13565,\n'
                                             '          "mean" : 938.106651,\n'
                                             '          "stddev" : 69.869104,\n'
                                             '          "N" : 2627701\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 1402,\n'
                                             '          "max" : 108673,\n'
                                             '          "mean" : 9369.322797,\n'
                                             '          "stddev" : '
                                             '2206.289705,\n'
                                             '          "N" : 2627704,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 7712,\n'
                                             '            "5.000000" : 8256,\n'
                                             '            "10.000000" : 8512,\n'
                                             '            "20.000000" : 8640,\n'
                                             '            "30.000000" : 8896,\n'
                                             '            "40.000000" : 8896,\n'
                                             '            "50.000000" : 9024,\n'
                                             '            "60.000000" : 9152,\n'
                                             '            "70.000000" : 9280,\n'
                                             '            "80.000000" : 9408,\n'
                                             '            "90.000000" : 9664,\n'
                                             '            "95.000000" : '
                                             '10688,\n'
                                             '            "99.000000" : '
                                             '20096,\n'
                                             '            "99.500000" : '
                                             '24192,\n'
                                             '            "99.900000" : '
                                             '29056,\n'
                                             '            "99.950000" : '
                                             '34048,\n'
                                             '            "99.990000" : 67072\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 7174,\n'
                                             '          "max" : 109625,\n'
                                             '          "mean" : '
                                             '10307.429430,\n'
                                             '          "stddev" : '
                                             '2207.091785,\n'
                                             '          "N" : 2627704\n'
                                             '        },\n'
                                             '        "bw_min" : 187307,\n'
                                             '        "bw_max" : 188201,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '187705.857143,\n'
                                             '        "bw_dev" : 270.564327,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 374614,\n'
                                             '        "iops_max" : 376402,\n'
                                             '        "iops_mean" : '
                                             '375411.785714,\n'
                                             '        "iops_stddev" : '
                                             '541.060523,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 99.985714,\n'
                                             '      "sys_cpu" : 0.000000,\n'
                                             '      "ctx" : 12,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.100000,\n'
                                             '        "2" : 3.338127,\n'
                                             '        "4" : 96.661683,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.010000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 93.003199,\n'
                                             '        "20" : 5.969895,\n'
                                             '        "50" : 0.989154,\n'
                                             '        "100" : 0.037714,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 4,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring_cmd-bdev_xnvme: '
                                             '(groupid=0, jobs=1): err= 0: '
                                             'pid=8655: Wed Oct 26 15:16:30 '
                                             '2022\n'
                                             '  read: IOPS=375k, BW=183MiB/s '
                                             '(192MB/s)(1283MiB/7001msec)\n'
                                             '    slat (nsec): min=651, '
                                             'max=13565, avg=938.11, '
                                             'stdev=69.87\n'
                                             '    clat (nsec): min=1402, '
                                             'max=108673, avg=9369.32, '
                                             'stdev=2206.29\n'
                                             '     lat (usec): min=7, max=109, '
                                             'avg=10.31, stdev= 2.21\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 7712],  '
                                             '5.00th=[ 8256], 10.00th=[ 8512], '
                                             '20.00th=[ 8640],\n'
                                             '     | 30.00th=[ 8896], '
                                             '40.00th=[ 8896], 50.00th=[ '
                                             '9024], 60.00th=[ 9152],\n'
                                             '     | 70.00th=[ 9280], '
                                             '80.00th=[ 9408], 90.00th=[ '
                                             '9664], 95.00th=[10688],\n'
                                             '     | 99.00th=[20096], '
                                             '99.50th=[24192], '
                                             '99.90th=[29056], '
                                             '99.95th=[34048],\n'
                                             '     | 99.99th=[67072]\n'
                                             '   bw (  KiB/s): min=187307, '
                                             'max=188201, per=100.00%, '
                                             'avg=187705.86, stdev=270.56, '
                                             'samples=14\n'
                                             '   iops        : min=374614, '
                                             'max=376402, avg=375411.79, '
                                             'stdev=541.06, samples=14\n'
                                             '  lat (usec)   : 2=0.01%, '
                                             '10=93.00%, 20=5.97%, 50=0.99%, '
                                             '100=0.04%\n'
                                             '  lat (usec)   : 250=0.01%\n'
                                             '  cpu          : usr=99.99%, '
                                             'sys=0.00%, ctx=12, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.1%, 2=3.3%, '
                                             '4=96.7%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=2627701,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=4\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=183MiB/s (192MB/s), '
                                             '183MiB/s-183MiB/s '
                                             '(192MB/s-192MB/s), io=1283MiB '
                                             '(1345MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_219.output'),
                                   'state': {'begin': 1666783100.123212,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_4_io_uring_cmd-bdev_xnvme_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8393714427947998,
                                             'end': 1666783100.9625835,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_219.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_219.state')},
                       'cmd_22': {'output': 'rm: cannot remove '
                                            "'/tmp/fio-output_512_1_io_uring_cmd-bdev_xnvme_1.txt': "
                                            'No such file or directory\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_22.output'),
                                  'state': {'begin': 1666782145.8041782,
                                            'cmd': 'rm '
                                                   '/tmp/fio-output_512_1_io_uring_cmd-bdev_xnvme_1.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.8044922351837158,
                                            'end': 1666782146.6086705,
                                            'err': 1,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_22.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_22.state')},
                       'cmd_220': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_512_4_io_uring_cmd-bdev_xnvme_1.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_220.output'),
                                   'state': {'begin': 1666783103.0728974,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_4_io_uring_cmd-bdev_xnvme_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8001556396484375,
                                             'end': 1666783103.873053,
                                             'err': 1,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_220.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_220.state')},
                       'cmd_221': {'output': 'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_221.output'),
                                   'state': {'begin': 1666783103.875193,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring_cmd-bdev_xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="4" --bs="512" '
                                                    '--direct="0" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="spdk_bdev" '
                                                    '--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" '
                                                    '--filename="Nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_4_io_uring_cmd-bdev_xnvme_1.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.461009740829468,
                                             'end': 1666783115.3362029,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_221.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_221.state')},
                       'cmd_222': {'output': 'io_uring_cmd-bdev_xnvme: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=spdk_bdev, iodepth=4\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666790205,\n'
                                             '  "timestamp_ms" : '
                                             '1666790205340,\n'
                                             '  "time" : "Wed Oct 26 15:16:45 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring_cmd-bdev_xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring_cmd-bdev_xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "4",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "0",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"spdk_bdev",\n'
                                             '        "spdk_conf" : '
                                             '"/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",\n'
                                             '        "filename" : "Nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '1416632832,\n'
                                             '        "io_kbytes" : 1383430,\n'
                                             '        "bw_bytes" : 202347212,\n'
                                             '        "bw" : 197604,\n'
                                             '        "iops" : 395208.970147,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 2766858,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 171,\n'
                                             '          "max" : 16481,\n'
                                             '          "mean" : 269.504734,\n'
                                             '          "stddev" : 26.227924,\n'
                                             '          "N" : 2766858\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 380,\n'
                                             '          "max" : 121328,\n'
                                             '          "mean" : 9537.189593,\n'
                                             '          "stddev" : '
                                             '2328.868613,\n'
                                             '          "N" : 2766861,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 6432,\n'
                                             '            "5.000000" : 7392,\n'
                                             '            "10.000000" : 8032,\n'
                                             '            "20.000000" : 8768,\n'
                                             '            "30.000000" : 9024,\n'
                                             '            "40.000000" : 9152,\n'
                                             '            "50.000000" : 9280,\n'
                                             '            "60.000000" : 9408,\n'
                                             '            "70.000000" : 9536,\n'
                                             '            "80.000000" : 9664,\n'
                                             '            "90.000000" : '
                                             '10304,\n'
                                             '            "95.000000" : '
                                             '11840,\n'
                                             '            "99.000000" : '
                                             '20096,\n'
                                             '            "99.500000" : '
                                             '21888,\n'
                                             '            "99.900000" : '
                                             '27264,\n'
                                             '            "99.950000" : '
                                             '33024,\n'
                                             '            "99.990000" : 68096\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 5581,\n'
                                             '          "max" : 121598,\n'
                                             '          "mean" : 9806.694310,\n'
                                             '          "stddev" : '
                                             '2330.569751,\n'
                                             '          "N" : 2766861\n'
                                             '        },\n'
                                             '        "bw_min" : 197016,\n'
                                             '        "bw_max" : 198564,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '197684.571429,\n'
                                             '        "bw_dev" : 480.434213,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 394032,\n'
                                             '        "iops_max" : 397128,\n'
                                             '        "iops_mean" : '
                                             '395369.142857,\n'
                                             '        "iops_stddev" : '
                                             '960.868426,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 99.985714,\n'
                                             '      "sys_cpu" : 0.000000,\n'
                                             '      "ctx" : 14,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.100000,\n'
                                             '        "2" : 12.759744,\n'
                                             '        "4" : 87.237184,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.010000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 87.316841,\n'
                                             '        "20" : 11.658495,\n'
                                             '        "50" : 0.982631,\n'
                                             '        "100" : 0.041672,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 4,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring_cmd-bdev_xnvme: '
                                             '(groupid=0, jobs=1): err= 0: '
                                             'pid=8715: Wed Oct 26 15:16:45 '
                                             '2022\n'
                                             '  read: IOPS=395k, BW=193MiB/s '
                                             '(202MB/s)(1351MiB/7001msec)\n'
                                             '    slat (nsec): min=171, '
                                             'max=16481, avg=269.50, '
                                             'stdev=26.23\n'
                                             '    clat (nsec): min=380, '
                                             'max=121328, avg=9537.19, '
                                             'stdev=2328.87\n'
                                             '     lat (usec): min=5, max=121, '
                                             'avg= 9.81, stdev= 2.33\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 6432],  '
                                             '5.00th=[ 7392], 10.00th=[ 8032], '
                                             '20.00th=[ 8768],\n'
                                             '     | 30.00th=[ 9024], '
                                             '40.00th=[ 9152], 50.00th=[ '
                                             '9280], 60.00th=[ 9408],\n'
                                             '     | 70.00th=[ 9536], '
                                             '80.00th=[ 9664], '
                                             '90.00th=[10304], '
                                             '95.00th=[11840],\n'
                                             '     | 99.00th=[20096], '
                                             '99.50th=[21888], '
                                             '99.90th=[27264], '
                                             '99.95th=[33024],\n'
                                             '     | 99.99th=[68096]\n'
                                             '   bw (  KiB/s): min=197016, '
                                             'max=198564, per=100.00%, '
                                             'avg=197684.57, stdev=480.43, '
                                             'samples=14\n'
                                             '   iops        : min=394032, '
                                             'max=397128, avg=395369.14, '
                                             'stdev=960.87, samples=14\n'
                                             '  lat (nsec)   : 500=0.01%\n'
                                             '  lat (usec)   : 10=87.32%, '
                                             '20=11.66%, 50=0.98%, 100=0.04%, '
                                             '250=0.01%\n'
                                             '  cpu          : usr=99.99%, '
                                             'sys=0.00%, ctx=14, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.1%, '
                                             '2=12.8%, 4=87.2%, 8=0.0%, '
                                             '16=0.0%, 32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=2766858,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=4\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=193MiB/s (202MB/s), '
                                             '193MiB/s-193MiB/s '
                                             '(202MB/s-202MB/s), io=1351MiB '
                                             '(1417MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_222.output'),
                                   'state': {'begin': 1666783115.3390436,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_4_io_uring_cmd-bdev_xnvme_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7917187213897705,
                                             'end': 1666783116.1307623,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_222.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_222.state')},
                       'cmd_223': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_512_4_io_uring_cmd-bdev_xnvme_2.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_223.output'),
                                   'state': {'begin': 1666783118.236984,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_4_io_uring_cmd-bdev_xnvme_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8149440288543701,
                                             'end': 1666783119.051928,
                                             'err': 1,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_223.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_223.state')},
                       'cmd_224': {'output': 'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_224.output'),
                                   'state': {'begin': 1666783119.0544753,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring_cmd-bdev_xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="4" --bs="512" '
                                                    '--direct="0" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="spdk_bdev" '
                                                    '--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" '
                                                    '--filename="Nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_4_io_uring_cmd-bdev_xnvme_2.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.496715068817139,
                                             'end': 1666783130.5511904,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_224.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_224.state')},
                       'cmd_225': {'output': 'io_uring_cmd-bdev_xnvme: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=spdk_bdev, iodepth=4\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666790220,\n'
                                             '  "timestamp_ms" : '
                                             '1666790220567,\n'
                                             '  "time" : "Wed Oct 26 15:17:00 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring_cmd-bdev_xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring_cmd-bdev_xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "4",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "0",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"spdk_bdev",\n'
                                             '        "spdk_conf" : '
                                             '"/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",\n'
                                             '        "filename" : "Nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '1625583104,\n'
                                             '        "io_kbytes" : 1587483,\n'
                                             '        "bw_bytes" : 232192987,\n'
                                             '        "bw" : 226750,\n'
                                             '        "iops" : 453501.499786,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 3174964,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 200,\n'
                                             '          "max" : 19697,\n'
                                             '          "mean" : 283.869127,\n'
                                             '          "stddev" : 75.463763,\n'
                                             '          "N" : 3174964\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 501,\n'
                                             '          "max" : 111770,\n'
                                             '          "mean" : 8226.494525,\n'
                                             '          "stddev" : '
                                             '2228.476383,\n'
                                             '          "N" : 3174967,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 6368,\n'
                                             '            "5.000000" : 6752,\n'
                                             '            "10.000000" : 7008,\n'
                                             '            "20.000000" : 7456,\n'
                                             '            "30.000000" : 7584,\n'
                                             '            "40.000000" : 7648,\n'
                                             '            "50.000000" : 7776,\n'
                                             '            "60.000000" : 7904,\n'
                                             '            "70.000000" : 8096,\n'
                                             '            "80.000000" : 8512,\n'
                                             '            "90.000000" : 9280,\n'
                                             '            "95.000000" : '
                                             '10176,\n'
                                             '            "99.000000" : '
                                             '18816,\n'
                                             '            "99.500000" : '
                                             '21376,\n'
                                             '            "99.900000" : '
                                             '27520,\n'
                                             '            "99.950000" : '
                                             '33536,\n'
                                             '            "99.990000" : 67072\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 5861,\n'
                                             '          "max" : 112050,\n'
                                             '          "mean" : 8510.363694,\n'
                                             '          "stddev" : '
                                             '2229.327804,\n'
                                             '          "N" : 3174967\n'
                                             '        },\n'
                                             '        "bw_min" : 226564,\n'
                                             '        "bw_max" : 227133,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '226841.785714,\n'
                                             '        "bw_dev" : 143.797496,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 453128,\n'
                                             '        "iops_max" : 454266,\n'
                                             '        "iops_mean" : '
                                             '453683.642857,\n'
                                             '        "iops_stddev" : '
                                             '287.590416,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 99.985714,\n'
                                             '      "sys_cpu" : 0.000000,\n'
                                             '      "ctx" : 10,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.100000,\n'
                                             '        "2" : 1.998952,\n'
                                             '        "4" : 97.991001,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.010000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 94.702460,\n'
                                             '        "20" : 4.585154,\n'
                                             '        "50" : 0.674307,\n'
                                             '        "100" : 0.037197,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 4,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring_cmd-bdev_xnvme: '
                                             '(groupid=0, jobs=1): err= 0: '
                                             'pid=8773: Wed Oct 26 15:17:00 '
                                             '2022\n'
                                             '  read: IOPS=454k, BW=221MiB/s '
                                             '(232MB/s)(1550MiB/7001msec)\n'
                                             '    slat (nsec): min=200, '
                                             'max=19697, avg=283.87, '
                                             'stdev=75.46\n'
                                             '    clat (nsec): min=501, '
                                             'max=111770, avg=8226.49, '
                                             'stdev=2228.48\n'
                                             '     lat (usec): min=5, max=112, '
                                             'avg= 8.51, stdev= 2.23\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 6368],  '
                                             '5.00th=[ 6752], 10.00th=[ 7008], '
                                             '20.00th=[ 7456],\n'
                                             '     | 30.00th=[ 7584], '
                                             '40.00th=[ 7648], 50.00th=[ '
                                             '7776], 60.00th=[ 7904],\n'
                                             '     | 70.00th=[ 8096], '
                                             '80.00th=[ 8512], 90.00th=[ '
                                             '9280], 95.00th=[10176],\n'
                                             '     | 99.00th=[18816], '
                                             '99.50th=[21376], '
                                             '99.90th=[27520], '
                                             '99.95th=[33536],\n'
                                             '     | 99.99th=[67072]\n'
                                             '   bw (  KiB/s): min=226564, '
                                             'max=227133, per=100.00%, '
                                             'avg=226841.79, stdev=143.80, '
                                             'samples=14\n'
                                             '   iops        : min=453128, '
                                             'max=454266, avg=453683.64, '
                                             'stdev=287.59, samples=14\n'
                                             '  lat (nsec)   : 750=0.01%\n'
                                             '  lat (usec)   : 10=94.70%, '
                                             '20=4.59%, 50=0.67%, 100=0.04%, '
                                             '250=0.01%\n'
                                             '  cpu          : usr=99.99%, '
                                             'sys=0.00%, ctx=10, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.1%, 2=2.0%, '
                                             '4=98.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=3174964,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=4\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=221MiB/s (232MB/s), '
                                             '221MiB/s-221MiB/s '
                                             '(232MB/s-232MB/s), io=1550MiB '
                                             '(1626MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_225.output'),
                                   'state': {'begin': 1666783130.5537598,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_4_io_uring_cmd-bdev_xnvme_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8054802417755127,
                                             'end': 1666783131.35924,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_225.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_225.state')},
                       'cmd_226': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_512_4_io_uring-reference_0.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_226.output'),
                                   'state': {'begin': 1666783132.3794498,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_4_io_uring-reference_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8866932392120361,
                                             'end': 1666783133.266143,
                                             'err': 1,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_226.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_226.state')},
                       'cmd_227': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_227.output'),
                                   'state': {'begin': 1666783133.2681544,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring-reference" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="4" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="io_uring" '
                                                    '--filename="/dev/nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_4_io_uring-reference_0.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.052165508270264,
                                             'end': 1666783144.32032,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_227.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_227.state')},
                       'cmd_228': {'output': 'io_uring-reference: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=io_uring, iodepth=4\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666790234,\n'
                                             '  "timestamp_ms" : '
                                             '1666790234399,\n'
                                             '  "time" : "Wed Oct 26 15:17:14 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring-reference",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring-reference",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "4",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"io_uring",\n'
                                             '        "filename" : '
                                             '"/dev/nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '1530035200,\n'
                                             '        "io_kbytes" : 1494175,\n'
                                             '        "bw_bytes" : 218545236,\n'
                                             '        "bw" : 213423,\n'
                                             '        "iops" : 426845.593487,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 2988346,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1313,\n'
                                             '          "max" : 23915,\n'
                                             '          "mean" : 1844.950892,\n'
                                             '          "stddev" : '
                                             '353.846891,\n'
                                             '          "N" : 2988347\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 110,\n'
                                             '          "max" : 126256,\n'
                                             '          "mean" : 7246.118301,\n'
                                             '          "stddev" : '
                                             '2909.869603,\n'
                                             '          "N" : 2988349,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 4192,\n'
                                             '            "5.000000" : 4320,\n'
                                             '            "10.000000" : 4768,\n'
                                             '            "20.000000" : 6240,\n'
                                             '            "30.000000" : 6496,\n'
                                             '            "40.000000" : 6560,\n'
                                             '            "50.000000" : 6752,\n'
                                             '            "60.000000" : 7072,\n'
                                             '            "70.000000" : 7520,\n'
                                             '            "80.000000" : 8256,\n'
                                             '            "90.000000" : 8768,\n'
                                             '            "95.000000" : 9408,\n'
                                             '            "99.000000" : '
                                             '20608,\n'
                                             '            "99.500000" : '
                                             '24960,\n'
                                             '            "99.900000" : '
                                             '27008,\n'
                                             '            "99.950000" : '
                                             '33024,\n'
                                             '            "99.990000" : 68096\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 5380,\n'
                                             '          "max" : 127619,\n'
                                             '          "mean" : 9091.069296,\n'
                                             '          "stddev" : '
                                             '2889.364715,\n'
                                             '          "N" : 2988349\n'
                                             '        },\n'
                                             '        "bw_min" : 212813,\n'
                                             '        "bw_max" : 214359,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '213525.857143,\n'
                                             '        "bw_dev" : 374.802688,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 425627,\n'
                                             '        "iops_max" : 428718,\n'
                                             '        "iops_mean" : '
                                             '427051.928571,\n'
                                             '        "iops_stddev" : '
                                             '749.479559,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 17.014286,\n'
                                             '      "sys_cpu" : 78.814286,\n'
                                             '      "ctx" : 36626,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.010000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.010000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 95.847569,\n'
                                             '        "20" : 2.632359,\n'
                                             '        "50" : 1.478845,\n'
                                             '        "100" : 0.039487,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 4,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ],\n'
                                             '  "disk_util" : [\n'
                                             '    {\n'
                                             '      "name" : "nvme0n1",\n'
                                             '      "read_ios" : 4302124,\n'
                                             '      "write_ios" : 0,\n'
                                             '      "read_merges" : 0,\n'
                                             '      "write_merges" : 0,\n'
                                             '      "read_ticks" : 23026,\n'
                                             '      "write_ticks" : 0,\n'
                                             '      "in_queue" : 23026,\n'
                                             '      "util" : 99.088748\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring-reference: (groupid=0, '
                                             'jobs=1): err= 0: pid=8823: Wed '
                                             'Oct 26 15:17:14 2022\n'
                                             '  read: IOPS=427k, BW=208MiB/s '
                                             '(219MB/s)(1459MiB/7001msec)\n'
                                             '    slat (nsec): min=1313, '
                                             'max=23915, avg=1844.95, '
                                             'stdev=353.85\n'
                                             '    clat (nsec): min=110, '
                                             'max=126256, avg=7246.12, '
                                             'stdev=2909.87\n'
                                             '     lat (usec): min=5, max=127, '
                                             'avg= 9.09, stdev= 2.89\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 4192],  '
                                             '5.00th=[ 4320], 10.00th=[ 4768], '
                                             '20.00th=[ 6240],\n'
                                             '     | 30.00th=[ 6496], '
                                             '40.00th=[ 6560], 50.00th=[ '
                                             '6752], 60.00th=[ 7072],\n'
                                             '     | 70.00th=[ 7520], '
                                             '80.00th=[ 8256], 90.00th=[ '
                                             '8768], 95.00th=[ 9408],\n'
                                             '     | 99.00th=[20608], '
                                             '99.50th=[24960], '
                                             '99.90th=[27008], '
                                             '99.95th=[33024],\n'
                                             '     | 99.99th=[68096]\n'
                                             '   bw (  KiB/s): min=212813, '
                                             'max=214359, per=100.00%, '
                                             'avg=213525.86, stdev=374.80, '
                                             'samples=14\n'
                                             '   iops        : min=425627, '
                                             'max=428718, avg=427051.93, '
                                             'stdev=749.48, samples=14\n'
                                             '  lat (nsec)   : 250=0.01%, '
                                             '500=0.01%\n'
                                             '  lat (usec)   : 2=0.01%, '
                                             '4=0.01%, 10=95.85%, 20=2.63%, '
                                             '50=1.48%\n'
                                             '  lat (usec)   : 100=0.04%, '
                                             '250=0.01%\n'
                                             '  cpu          : usr=17.01%, '
                                             'sys=78.81%, ctx=36626, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, 2=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=2988346,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=4\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=208MiB/s (219MB/s), '
                                             '208MiB/s-208MiB/s '
                                             '(219MB/s-219MB/s), io=1459MiB '
                                             '(1530MB), run=7001-7001msec\n'
                                             '\n'
                                             'Disk stats (read/write):\n'
                                             '  nvme0n1: ios=4302124/0, '
                                             'merge=0/0, ticks=23026/0, '
                                             'in_queue=23026, util=99.09%\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_228.output'),
                                   'state': {'begin': 1666783144.3229122,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_4_io_uring-reference_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8065760135650635,
                                             'end': 1666783145.1294882,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_228.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_228.state')},
                       'cmd_229': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_512_4_io_uring-reference_1.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_229.output'),
                                   'state': {'begin': 1666783146.164021,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_4_io_uring-reference_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.798626184463501,
                                             'end': 1666783146.9626472,
                                             'err': 1,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_229.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_229.state')},
                       'cmd_23': {'output': 'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_23.output'),
                                  'state': {'begin': 1666782146.6109998,
                                            'cmd': '/opt/fio/bin/fio '
                                                   '--name="io_uring_cmd-bdev_xnvme" '
                                                   '--rw="randread" '
                                                   '--size="12G" --iodepth="1" '
                                                   '--bs="512" --direct="0" '
                                                   '--thread="1" '
                                                   '--time_based="1" '
                                                   '--runtime="7" '
                                                   '--ramp_time="3" '
                                                   '--norandommap="1" '
                                                   '--allow_file_create="0" '
                                                   '--ioengine="spdk_bdev" '
                                                   '--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" '
                                                   '--filename="Nvme0n1" '
                                                   '--output-format="normal,json" '
                                                   '--output="/tmp/fio-output_512_1_io_uring_cmd-bdev_xnvme_1.txt" '
                                                   '--cpus_allowed="1"',
                                            'cwd': 'None',
                                            'elapsed': 11.522717714309692,
                                            'end': 1666782158.1337175,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_23.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_23.state')},
                       'cmd_230': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_230.output'),
                                   'state': {'begin': 1666783146.9648237,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring-reference" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="4" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="io_uring" '
                                                    '--filename="/dev/nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_4_io_uring-reference_1.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.045536041259766,
                                             'end': 1666783158.0103598,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_230.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_230.state')},
                       'cmd_231': {'output': 'io_uring-reference: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=io_uring, iodepth=4\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666790248,\n'
                                             '  "timestamp_ms" : '
                                             '1666790248087,\n'
                                             '  "time" : "Wed Oct 26 15:17:28 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring-reference",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring-reference",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "4",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"io_uring",\n'
                                             '        "filename" : '
                                             '"/dev/nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '1519099904,\n'
                                             '        "io_kbytes" : 1483496,\n'
                                             '        "bw_bytes" : 216983274,\n'
                                             '        "bw" : 211897,\n'
                                             '        "iops" : 423794.886445,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 2966988,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1322,\n'
                                             '          "max" : 22562,\n'
                                             '          "mean" : 1861.858082,\n'
                                             '          "stddev" : '
                                             '352.891983,\n'
                                             '          "N" : 2966989\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 100,\n'
                                             '          "max" : 122830,\n'
                                             '          "mean" : 7297.526278,\n'
                                             '          "stddev" : '
                                             '2887.510987,\n'
                                             '          "N" : 2966991,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 4192,\n'
                                             '            "5.000000" : 4320,\n'
                                             '            "10.000000" : 4832,\n'
                                             '            "20.000000" : 6368,\n'
                                             '            "30.000000" : 6560,\n'
                                             '            "40.000000" : 6624,\n'
                                             '            "50.000000" : 6880,\n'
                                             '            "60.000000" : 7200,\n'
                                             '            "70.000000" : 7520,\n'
                                             '            "80.000000" : 8256,\n'
                                             '            "90.000000" : 8768,\n'
                                             '            "95.000000" : 9408,\n'
                                             '            "99.000000" : '
                                             '20608,\n'
                                             '            "99.500000" : '
                                             '24960,\n'
                                             '            "99.900000" : '
                                             '27008,\n'
                                             '            "99.950000" : '
                                             '33024,\n'
                                             '            "99.990000" : 68096\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 5731,\n'
                                             '          "max" : 124212,\n'
                                             '          "mean" : 9159.384584,\n'
                                             '          "stddev" : '
                                             '2868.348094,\n'
                                             '          "N" : 2966991\n'
                                             '        },\n'
                                             '        "bw_min" : 211271,\n'
                                             '        "bw_max" : 212946,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '212027.500000,\n'
                                             '        "bw_dev" : 425.605225,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 422542,\n'
                                             '        "iops_max" : 425892,\n'
                                             '        "iops_mean" : '
                                             '424055.357143,\n'
                                             '        "iops_stddev" : '
                                             '851.333308,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 17.200000,\n'
                                             '      "sys_cpu" : 78.642857,\n'
                                             '      "ctx" : 35976,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.010000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 95.813296,\n'
                                             '        "20" : 2.761386,\n'
                                             '        "50" : 1.384872,\n'
                                             '        "100" : 0.039299,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 4,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ],\n'
                                             '  "disk_util" : [\n'
                                             '    {\n'
                                             '      "name" : "nvme0n1",\n'
                                             '      "read_ios" : 4273785,\n'
                                             '      "write_ios" : 0,\n'
                                             '      "read_merges" : 0,\n'
                                             '      "write_merges" : 0,\n'
                                             '      "read_ticks" : 22940,\n'
                                             '      "write_ticks" : 0,\n'
                                             '      "in_queue" : 22940,\n'
                                             '      "util" : 99.098564\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring-reference: (groupid=0, '
                                             'jobs=1): err= 0: pid=8869: Wed '
                                             'Oct 26 15:17:28 2022\n'
                                             '  read: IOPS=424k, BW=207MiB/s '
                                             '(217MB/s)(1449MiB/7001msec)\n'
                                             '    slat (nsec): min=1322, '
                                             'max=22562, avg=1861.86, '
                                             'stdev=352.89\n'
                                             '    clat (nsec): min=100, '
                                             'max=122830, avg=7297.53, '
                                             'stdev=2887.51\n'
                                             '     lat (usec): min=5, max=124, '
                                             'avg= 9.16, stdev= 2.87\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 4192],  '
                                             '5.00th=[ 4320], 10.00th=[ 4832], '
                                             '20.00th=[ 6368],\n'
                                             '     | 30.00th=[ 6560], '
                                             '40.00th=[ 6624], 50.00th=[ '
                                             '6880], 60.00th=[ 7200],\n'
                                             '     | 70.00th=[ 7520], '
                                             '80.00th=[ 8256], 90.00th=[ '
                                             '8768], 95.00th=[ 9408],\n'
                                             '     | 99.00th=[20608], '
                                             '99.50th=[24960], '
                                             '99.90th=[27008], '
                                             '99.95th=[33024],\n'
                                             '     | 99.99th=[68096]\n'
                                             '   bw (  KiB/s): min=211271, '
                                             'max=212946, per=100.00%, '
                                             'avg=212027.50, stdev=425.61, '
                                             'samples=14\n'
                                             '   iops        : min=422542, '
                                             'max=425892, avg=424055.36, '
                                             'stdev=851.33, samples=14\n'
                                             '  lat (nsec)   : 250=0.01%\n'
                                             '  lat (usec)   : 2=0.01%, '
                                             '4=0.01%, 10=95.81%, 20=2.76%, '
                                             '50=1.38%\n'
                                             '  lat (usec)   : 100=0.04%, '
                                             '250=0.01%\n'
                                             '  cpu          : usr=17.20%, '
                                             'sys=78.64%, ctx=35976, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, 2=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=2966988,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=4\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=207MiB/s (217MB/s), '
                                             '207MiB/s-207MiB/s '
                                             '(217MB/s-217MB/s), io=1449MiB '
                                             '(1519MB), run=7001-7001msec\n'
                                             '\n'
                                             'Disk stats (read/write):\n'
                                             '  nvme0n1: ios=4273785/0, '
                                             'merge=0/0, ticks=22940/0, '
                                             'in_queue=22940, util=99.10%\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_231.output'),
                                   'state': {'begin': 1666783158.0133226,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_4_io_uring-reference_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8554275035858154,
                                             'end': 1666783158.86875,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_231.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_231.state')},
                       'cmd_232': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_512_4_io_uring-reference_2.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_232.output'),
                                   'state': {'begin': 1666783159.9105759,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_4_io_uring-reference_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7936725616455078,
                                             'end': 1666783160.7042484,
                                             'err': 1,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_232.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_232.state')},
                       'cmd_233': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_233.output'),
                                   'state': {'begin': 1666783160.7061718,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring-reference" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="4" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="io_uring" '
                                                    '--filename="/dev/nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_4_io_uring-reference_2.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.07721996307373,
                                             'end': 1666783171.7833917,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_233.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_233.state')},
                       'cmd_234': {'output': 'io_uring-reference: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=io_uring, iodepth=4\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666790261,\n'
                                             '  "timestamp_ms" : '
                                             '1666790261828,\n'
                                             '  "time" : "Wed Oct 26 15:17:41 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring-reference",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring-reference",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "4",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"io_uring",\n'
                                             '        "filename" : '
                                             '"/dev/nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '1502555648,\n'
                                             '        "io_kbytes" : 1467339,\n'
                                             '        "bw_bytes" : 214620146,\n'
                                             '        "bw" : 209589,\n'
                                             '        "iops" : 419179.402942,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 2934675,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1333,\n'
                                             '          "max" : 17603,\n'
                                             '          "mean" : 1865.526482,\n'
                                             '          "stddev" : '
                                             '355.622282,\n'
                                             '          "N" : 2934676\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 100,\n'
                                             '          "max" : 118191,\n'
                                             '          "mean" : 7382.369814,\n'
                                             '          "stddev" : '
                                             '2876.847571,\n'
                                             '          "N" : 2934678,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 4256,\n'
                                             '            "5.000000" : 4320,\n'
                                             '            "10.000000" : 4896,\n'
                                             '            "20.000000" : 6496,\n'
                                             '            "30.000000" : 6624,\n'
                                             '            "40.000000" : 6752,\n'
                                             '            "50.000000" : 7136,\n'
                                             '            "60.000000" : 7264,\n'
                                             '            "70.000000" : 7648,\n'
                                             '            "80.000000" : 8256,\n'
                                             '            "90.000000" : 8640,\n'
                                             '            "95.000000" : 9408,\n'
                                             '            "99.000000" : '
                                             '20864,\n'
                                             '            "99.500000" : '
                                             '25216,\n'
                                             '            "99.900000" : '
                                             '27008,\n'
                                             '            "99.950000" : '
                                             '33024,\n'
                                             '            "99.990000" : 68096\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 5760,\n'
                                             '          "max" : 119564,\n'
                                             '          "mean" : 9247.896274,\n'
                                             '          "stddev" : '
                                             '2865.318682,\n'
                                             '          "N" : 2934678\n'
                                             '        },\n'
                                             '        "bw_min" : 208949,\n'
                                             '        "bw_max" : 210632,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '209732.785714,\n'
                                             '        "bw_dev" : 576.971694,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 417898,\n'
                                             '        "iops_max" : 421264,\n'
                                             '        "iops_mean" : '
                                             '419465.785714,\n'
                                             '        "iops_stddev" : '
                                             '1153.870889,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 17.242857,\n'
                                             '      "sys_cpu" : 78.757143,\n'
                                             '      "ctx" : 32783,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 95.971786,\n'
                                             '        "20" : 2.527912,\n'
                                             '        "50" : 1.462308,\n'
                                             '        "100" : 0.035949,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 4,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ],\n'
                                             '  "disk_util" : [\n'
                                             '    {\n'
                                             '      "name" : "nvme0n1",\n'
                                             '      "read_ios" : 4230620,\n'
                                             '      "write_ios" : 0,\n'
                                             '      "read_merges" : 0,\n'
                                             '      "write_merges" : 0,\n'
                                             '      "read_ticks" : 22672,\n'
                                             '      "write_ticks" : 0,\n'
                                             '      "in_queue" : 22672,\n'
                                             '      "util" : 99.098564\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring-reference: (groupid=0, '
                                             'jobs=1): err= 0: pid=8917: Wed '
                                             'Oct 26 15:17:41 2022\n'
                                             '  read: IOPS=419k, BW=205MiB/s '
                                             '(215MB/s)(1433MiB/7001msec)\n'
                                             '    slat (nsec): min=1333, '
                                             'max=17603, avg=1865.53, '
                                             'stdev=355.62\n'
                                             '    clat (nsec): min=100, '
                                             'max=118191, avg=7382.37, '
                                             'stdev=2876.85\n'
                                             '     lat (usec): min=5, max=119, '
                                             'avg= 9.25, stdev= 2.87\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 4256],  '
                                             '5.00th=[ 4320], 10.00th=[ 4896], '
                                             '20.00th=[ 6496],\n'
                                             '     | 30.00th=[ 6624], '
                                             '40.00th=[ 6752], 50.00th=[ '
                                             '7136], 60.00th=[ 7264],\n'
                                             '     | 70.00th=[ 7648], '
                                             '80.00th=[ 8256], 90.00th=[ '
                                             '8640], 95.00th=[ 9408],\n'
                                             '     | 99.00th=[20864], '
                                             '99.50th=[25216], '
                                             '99.90th=[27008], '
                                             '99.95th=[33024],\n'
                                             '     | 99.99th=[68096]\n'
                                             '   bw (  KiB/s): min=208949, '
                                             'max=210632, per=100.00%, '
                                             'avg=209732.79, stdev=576.97, '
                                             'samples=14\n'
                                             '   iops        : min=417898, '
                                             'max=421264, avg=419465.79, '
                                             'stdev=1153.87, samples=14\n'
                                             '  lat (nsec)   : 250=0.01%\n'
                                             '  lat (usec)   : 4=0.01%, '
                                             '10=95.97%, 20=2.53%, 50=1.46%, '
                                             '100=0.04%\n'
                                             '  lat (usec)   : 250=0.01%\n'
                                             '  cpu          : usr=17.24%, '
                                             'sys=78.76%, ctx=32783, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, 2=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=2934675,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=4\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=205MiB/s (215MB/s), '
                                             '205MiB/s-205MiB/s '
                                             '(215MB/s-215MB/s), io=1433MiB '
                                             '(1503MB), run=7001-7001msec\n'
                                             '\n'
                                             'Disk stats (read/write):\n'
                                             '  nvme0n1: ios=4230620/0, '
                                             'merge=0/0, ticks=22672/0, '
                                             'in_queue=22672, util=99.10%\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_234.output'),
                                   'state': {'begin': 1666783171.7861347,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_4_io_uring-reference_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7979645729064941,
                                             'end': 1666783172.5840993,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_234.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_234.state')},
                       'cmd_235': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_512_4_io_uring-bdev_uring_0.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_235.output'),
                                   'state': {'begin': 1666783174.6949475,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_4_io_uring-bdev_uring_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7767727375030518,
                                             'end': 1666783175.4717202,
                                             'err': 1,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_235.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_235.state')},
                       'cmd_236': {'output': 'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_236.output'),
                                   'state': {'begin': 1666783175.4740148,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring-bdev_uring" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="4" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="spdk_bdev" '
                                                    '--spdk_conf="/tmp/spdk_bdev_uring_foo.conf" '
                                                    '--filename="Nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_4_io_uring-bdev_uring_0.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.314560413360596,
                                             'end': 1666783186.7885752,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_236.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_236.state')},
                       'cmd_237': {'output': 'io_uring-bdev_uring: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=spdk_bdev, iodepth=4\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666790276,\n'
                                             '  "timestamp_ms" : '
                                             '1666790276804,\n'
                                             '  "time" : "Wed Oct 26 15:17:56 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring-bdev_uring",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring-bdev_uring",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "4",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"spdk_bdev",\n'
                                             '        "spdk_conf" : '
                                             '"/tmp/spdk_bdev_uring_foo.conf",\n'
                                             '        "filename" : "Nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '1205324288,\n'
                                             '        "io_kbytes" : 1177074,\n'
                                             '        "bw_bytes" : 172164589,\n'
                                             '        "bw" : 168129,\n'
                                             '        "iops" : 336258.534495,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 2354146,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 110,\n'
                                             '          "max" : 17002,\n'
                                             '          "mean" : 179.156082,\n'
                                             '          "stddev" : '
                                             '149.002890,\n'
                                             '          "N" : 2354146\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 5971,\n'
                                             '          "max" : 115316,\n'
                                             '          "mean" : '
                                             '11061.462771,\n'
                                             '          "stddev" : '
                                             '3262.876327,\n'
                                             '          "N" : 2354147,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 6496,\n'
                                             '            "5.000000" : 7328,\n'
                                             '            "10.000000" : 7968,\n'
                                             '            "20.000000" : 8384,\n'
                                             '            "30.000000" : 8768,\n'
                                             '            "40.000000" : 9152,\n'
                                             '            "50.000000" : '
                                             '10432,\n'
                                             '            "60.000000" : '
                                             '12096,\n'
                                             '            "70.000000" : '
                                             '12736,\n'
                                             '            "80.000000" : '
                                             '13248,\n'
                                             '            "90.000000" : '
                                             '13760,\n'
                                             '            "95.000000" : '
                                             '17280,\n'
                                             '            "99.000000" : '
                                             '21376,\n'
                                             '            "99.500000" : '
                                             '24192,\n'
                                             '            "99.900000" : '
                                             '30336,\n'
                                             '            "99.950000" : '
                                             '35072,\n'
                                             '            "99.990000" : 66048\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 6171,\n'
                                             '          "max" : 115496,\n'
                                             '          "mean" : '
                                             '11240.618823,\n'
                                             '          "stddev" : '
                                             '3234.373994,\n'
                                             '          "N" : 2354147\n'
                                             '        },\n'
                                             '        "bw_min" : 166257,\n'
                                             '        "bw_max" : 170589,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '168231.142857,\n'
                                             '        "bw_dev" : 1493.829661,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 332515,\n'
                                             '        "iops_max" : 341178,\n'
                                             '        "iops_mean" : '
                                             '336462.428571,\n'
                                             '        "iops_stddev" : '
                                             '2987.544238,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 41.185714,\n'
                                             '      "sys_cpu" : 58.785714,\n'
                                             '      "ctx" : 11,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 6.693255,\n'
                                             '        "2" : 43.215119,\n'
                                             '        "4" : 50.091626,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 47.577253,\n'
                                             '        "20" : 51.221589,\n'
                                             '        "50" : 1.163182,\n'
                                             '        "100" : 0.037848,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 4,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring-bdev_uring: (groupid=0, '
                                             'jobs=1): err= 0: pid=8974: Wed '
                                             'Oct 26 15:17:56 2022\n'
                                             '  read: IOPS=336k, BW=164MiB/s '
                                             '(172MB/s)(1149MiB/7001msec)\n'
                                             '    slat (nsec): min=110, '
                                             'max=17002, avg=179.16, '
                                             'stdev=149.00\n'
                                             '    clat (usec): min=5, max=115, '
                                             'avg=11.06, stdev= 3.26\n'
                                             '     lat (usec): min=6, max=115, '
                                             'avg=11.24, stdev= 3.23\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 6496],  '
                                             '5.00th=[ 7328], 10.00th=[ 7968], '
                                             '20.00th=[ 8384],\n'
                                             '     | 30.00th=[ 8768], '
                                             '40.00th=[ 9152], '
                                             '50.00th=[10432], '
                                             '60.00th=[12096],\n'
                                             '     | 70.00th=[12736], '
                                             '80.00th=[13248], '
                                             '90.00th=[13760], '
                                             '95.00th=[17280],\n'
                                             '     | 99.00th=[21376], '
                                             '99.50th=[24192], '
                                             '99.90th=[30336], '
                                             '99.95th=[35072],\n'
                                             '     | 99.99th=[66048]\n'
                                             '   bw (  KiB/s): min=166257, '
                                             'max=170589, per=100.00%, '
                                             'avg=168231.14, stdev=1493.83, '
                                             'samples=14\n'
                                             '   iops        : min=332515, '
                                             'max=341178, avg=336462.43, '
                                             'stdev=2987.54, samples=14\n'
                                             '  lat (usec)   : 10=47.58%, '
                                             '20=51.22%, 50=1.16%, 100=0.04%, '
                                             '250=0.01%\n'
                                             '  cpu          : usr=41.19%, '
                                             'sys=58.79%, ctx=11, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=6.7%, '
                                             '2=43.2%, 4=50.1%, 8=0.0%, '
                                             '16=0.0%, 32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=2354146,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=4\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=164MiB/s (172MB/s), '
                                             '164MiB/s-164MiB/s '
                                             '(172MB/s-172MB/s), io=1149MiB '
                                             '(1205MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_237.output'),
                                   'state': {'begin': 1666783186.7954512,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_4_io_uring-bdev_uring_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.816450834274292,
                                             'end': 1666783187.611902,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_237.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_237.state')},
                       'cmd_238': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_512_4_io_uring-bdev_uring_1.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_238.output'),
                                   'state': {'begin': 1666783189.7250276,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_4_io_uring-bdev_uring_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7946679592132568,
                                             'end': 1666783190.5196955,
                                             'err': 1,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_238.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_238.state')},
                       'cmd_239': {'output': 'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_239.output'),
                                   'state': {'begin': 1666783190.5224497,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring-bdev_uring" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="4" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="spdk_bdev" '
                                                    '--spdk_conf="/tmp/spdk_bdev_uring_foo.conf" '
                                                    '--filename="Nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_4_io_uring-bdev_uring_1.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.44857120513916,
                                             'end': 1666783201.971021,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_239.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_239.state')},
                       'cmd_24': {'output': 'io_uring_cmd-bdev_xnvme: (g=0): '
                                            'rw=randread, bs=(R) 512B-512B, '
                                            '(W) 512B-512B, (T) 512B-512B, '
                                            'ioengine=spdk_bdev, iodepth=1\n'
                                            'fio-3.32-dirty\n'
                                            'Starting 1 thread\n'
                                            '{\n'
                                            '  "fio version" : '
                                            '"fio-3.32-dirty",\n'
                                            '  "timestamp" : 1666789248,\n'
                                            '  "timestamp_ms" : '
                                            '1666789248165,\n'
                                            '  "time" : "Wed Oct 26 15:00:48 '
                                            '2022",\n'
                                            '  "jobs" : [\n'
                                            '    {\n'
                                            '      "jobname" : '
                                            '"io_uring_cmd-bdev_xnvme",\n'
                                            '      "groupid" : 0,\n'
                                            '      "error" : 0,\n'
                                            '      "eta" : 0,\n'
                                            '      "elapsed" : 11,\n'
                                            '      "job options" : {\n'
                                            '        "name" : '
                                            '"io_uring_cmd-bdev_xnvme",\n'
                                            '        "rw" : "randread",\n'
                                            '        "size" : "12G",\n'
                                            '        "iodepth" : "1",\n'
                                            '        "bs" : "512",\n'
                                            '        "direct" : "0",\n'
                                            '        "thread" : "1",\n'
                                            '        "time_based" : "1",\n'
                                            '        "runtime" : "7",\n'
                                            '        "ramp_time" : "3",\n'
                                            '        "norandommap" : "1",\n'
                                            '        "allow_file_create" : '
                                            '"0",\n'
                                            '        "ioengine" : '
                                            '"spdk_bdev",\n'
                                            '        "spdk_conf" : '
                                            '"/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",\n'
                                            '        "filename" : "Nvme0n1",\n'
                                            '        "cpus_allowed" : "1"\n'
                                            '      },\n'
                                            '      "read" : {\n'
                                            '        "io_bytes" : 407997952,\n'
                                            '        "io_kbytes" : 398435,\n'
                                            '        "bw_bytes" : 58277096,\n'
                                            '        "bw" : 56911,\n'
                                            '        "iops" : 113822.453935,\n'
                                            '        "runtime" : 7001,\n'
                                            '        "total_ios" : 796871,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 711,\n'
                                            '          "max" : 20899,\n'
                                            '          "mean" : 921.766521,\n'
                                            '          "stddev" : 69.333618,\n'
                                            '          "N" : 796871\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 5120,\n'
                                            '          "max" : 80270,\n'
                                            '          "mean" : 7528.237944,\n'
                                            '          "stddev" : '
                                            '1123.122707,\n'
                                            '          "N" : 796871,\n'
                                            '          "percentile" : {\n'
                                            '            "1.000000" : 6880,\n'
                                            '            "5.000000" : 7008,\n'
                                            '            "10.000000" : 7072,\n'
                                            '            "20.000000" : 7200,\n'
                                            '            "30.000000" : 7264,\n'
                                            '            "40.000000" : 7328,\n'
                                            '            "50.000000" : 7392,\n'
                                            '            "60.000000" : 7520,\n'
                                            '            "70.000000" : 7584,\n'
                                            '            "80.000000" : 7648,\n'
                                            '            "90.000000" : 7776,\n'
                                            '            "95.000000" : 7904,\n'
                                            '            "99.000000" : 11712,\n'
                                            '            "99.500000" : 14400,\n'
                                            '            "99.900000" : 16512,\n'
                                            '            "99.950000" : 17280,\n'
                                            '            "99.990000" : 60672\n'
                                            '          }\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 7153,\n'
                                            '          "max" : 81202,\n'
                                            '          "mean" : 8450.004465,\n'
                                            '          "stddev" : '
                                            '1126.519340,\n'
                                            '          "N" : 796871\n'
                                            '        },\n'
                                            '        "bw_min" : 56840,\n'
                                            '        "bw_max" : 57028,\n'
                                            '        "bw_agg" : 100.000000,\n'
                                            '        "bw_mean" : '
                                            '56925.714286,\n'
                                            '        "bw_dev" : 58.529977,\n'
                                            '        "bw_samples" : 14,\n'
                                            '        "iops_min" : 113680,\n'
                                            '        "iops_max" : 114056,\n'
                                            '        "iops_mean" : '
                                            '113851.500000,\n'
                                            '        "iops_stddev" : '
                                            '117.153909,\n'
                                            '        "iops_samples" : 14\n'
                                            '      },\n'
                                            '      "write" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "trim" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "sync" : {\n'
                                            '        "total_ios" : 0,\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        }\n'
                                            '      },\n'
                                            '      "job_runtime" : 7000,\n'
                                            '      "usr_cpu" : 99.985714,\n'
                                            '      "sys_cpu" : 0.000000,\n'
                                            '      "ctx" : 11,\n'
                                            '      "majf" : 0,\n'
                                            '      "minf" : 0,\n'
                                            '      "iodepth_level" : {\n'
                                            '        "1" : 100.000000,\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_submit" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_complete" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ns" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_us" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 98.200336,\n'
                                            '        "20" : 1.771805,\n'
                                            '        "50" : 0.010000,\n'
                                            '        "100" : 0.018322,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ms" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000,\n'
                                            '        "2000" : 0.000000,\n'
                                            '        ">=2000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_depth" : 1,\n'
                                            '      "latency_target" : 0,\n'
                                            '      "latency_percentile" : '
                                            '100.000000,\n'
                                            '      "latency_window" : 0\n'
                                            '    }\n'
                                            '  ]\n'
                                            '}\n'
                                            '\n'
                                            'io_uring_cmd-bdev_xnvme: '
                                            '(groupid=0, jobs=1): err= 0: '
                                            'pid=5285: Wed Oct 26 15:00:48 '
                                            '2022\n'
                                            '  read: IOPS=114k, BW=55.6MiB/s '
                                            '(58.3MB/s)(389MiB/7001msec)\n'
                                            '    slat (nsec): min=711, '
                                            'max=20899, avg=921.77, '
                                            'stdev=69.33\n'
                                            '    clat (nsec): min=5120, '
                                            'max=80270, avg=7528.24, '
                                            'stdev=1123.12\n'
                                            '     lat (nsec): min=7153, '
                                            'max=81202, avg=8450.00, '
                                            'stdev=1126.52\n'
                                            '    clat percentiles (nsec):\n'
                                            '     |  1.00th=[ 6880],  5.00th=[ '
                                            '7008], 10.00th=[ 7072], 20.00th=[ '
                                            '7200],\n'
                                            '     | 30.00th=[ 7264], 40.00th=[ '
                                            '7328], 50.00th=[ 7392], 60.00th=[ '
                                            '7520],\n'
                                            '     | 70.00th=[ 7584], 80.00th=[ '
                                            '7648], 90.00th=[ 7776], 95.00th=[ '
                                            '7904],\n'
                                            '     | 99.00th=[11712], '
                                            '99.50th=[14400], 99.90th=[16512], '
                                            '99.95th=[17280],\n'
                                            '     | 99.99th=[60672]\n'
                                            '   bw (  KiB/s): min=56840, '
                                            'max=57028, per=100.00%, '
                                            'avg=56925.71, stdev=58.53, '
                                            'samples=14\n'
                                            '   iops        : min=113680, '
                                            'max=114056, avg=113851.50, '
                                            'stdev=117.15, samples=14\n'
                                            '  lat (usec)   : 10=98.20%, '
                                            '20=1.77%, 50=0.01%, 100=0.02%\n'
                                            '  cpu          : usr=99.99%, '
                                            'sys=0.00%, ctx=11, majf=0, '
                                            'minf=0\n'
                                            '  IO depths    : 1=100.0%, '
                                            '2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, >=64=0.0%\n'
                                            '     submit    : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     complete  : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     issued rwts: '
                                            'total=796871,0,0,0 short=0,0,0,0 '
                                            'dropped=0,0,0,0\n'
                                            '     latency   : target=0, '
                                            'window=0, percentile=100.00%, '
                                            'depth=1\n'
                                            '\n'
                                            'Run status group 0 (all jobs):\n'
                                            '   READ: bw=55.6MiB/s (58.3MB/s), '
                                            '55.6MiB/s-55.6MiB/s '
                                            '(58.3MB/s-58.3MB/s), io=389MiB '
                                            '(408MB), run=7001-7001msec\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_24.output'),
                                  'state': {'begin': 1666782158.1400828,
                                            'cmd': 'cat '
                                                   '/tmp/fio-output_512_1_io_uring_cmd-bdev_xnvme_1.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.7942419052124023,
                                            'end': 1666782158.9343247,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_24.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_24.state')},
                       'cmd_240': {'output': 'io_uring-bdev_uring: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=spdk_bdev, iodepth=4\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666790291,\n'
                                             '  "timestamp_ms" : '
                                             '1666790291984,\n'
                                             '  "time" : "Wed Oct 26 15:18:11 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring-bdev_uring",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring-bdev_uring",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "4",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"spdk_bdev",\n'
                                             '        "spdk_conf" : '
                                             '"/tmp/spdk_bdev_uring_foo.conf",\n'
                                             '        "filename" : "Nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '1173137408,\n'
                                             '        "io_kbytes" : 1145642,\n'
                                             '        "bw_bytes" : 167591058,\n'
                                             '        "bw" : 163663,\n'
                                             '        "iops" : 327325.857143,\n'
                                             '        "runtime" : 7000,\n'
                                             '        "total_ios" : 2291281,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 120,\n'
                                             '          "max" : 15419,\n'
                                             '          "mean" : 174.303148,\n'
                                             '          "stddev" : '
                                             '129.262014,\n'
                                             '          "N" : 2291281\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 6031,\n'
                                             '          "max" : 97032,\n'
                                             '          "mean" : '
                                             '11358.835785,\n'
                                             '          "stddev" : '
                                             '3221.532516,\n'
                                             '          "N" : 2291284,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 6496,\n'
                                             '            "5.000000" : 7392,\n'
                                             '            "10.000000" : 8032,\n'
                                             '            "20.000000" : 8512,\n'
                                             '            "30.000000" : 8896,\n'
                                             '            "40.000000" : 9664,\n'
                                             '            "50.000000" : '
                                             '11712,\n'
                                             '            "60.000000" : '
                                             '12608,\n'
                                             '            "70.000000" : '
                                             '12992,\n'
                                             '            "80.000000" : '
                                             '13376,\n'
                                             '            "90.000000" : '
                                             '13760,\n'
                                             '            "95.000000" : '
                                             '17536,\n'
                                             '            "99.000000" : '
                                             '21376,\n'
                                             '            "99.500000" : '
                                             '23936,\n'
                                             '            "99.900000" : '
                                             '29568,\n'
                                             '            "99.950000" : '
                                             '35072,\n'
                                             '            "99.990000" : 66048\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 6202,\n'
                                             '          "max" : 97222,\n'
                                             '          "mean" : '
                                             '11533.138909,\n'
                                             '          "stddev" : '
                                             '3199.841005,\n'
                                             '          "N" : 2291284\n'
                                             '        },\n'
                                             '        "bw_min" : 161805,\n'
                                             '        "bw_max" : 165426,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '163730.928571,\n'
                                             '        "bw_dev" : 1280.287135,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 323611,\n'
                                             '        "iops_max" : 330852,\n'
                                             '        "iops_mean" : '
                                             '327461.928571,\n'
                                             '        "iops_stddev" : '
                                             '2560.458567,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 42.842857,\n'
                                             '      "sys_cpu" : 57.128571,\n'
                                             '      "ctx" : 16,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 9.001777,\n'
                                             '        "2" : 44.955944,\n'
                                             '        "4" : 46.042279,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 41.212099,\n'
                                             '        "20" : 57.586259,\n'
                                             '        "50" : 1.166116,\n'
                                             '        "100" : 0.035657,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 4,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring-bdev_uring: (groupid=0, '
                                             'jobs=1): err= 0: pid=9031: Wed '
                                             'Oct 26 15:18:11 2022\n'
                                             '  read: IOPS=327k, BW=160MiB/s '
                                             '(168MB/s)(1119MiB/7000msec)\n'
                                             '    slat (nsec): min=120, '
                                             'max=15419, avg=174.30, '
                                             'stdev=129.26\n'
                                             '    clat (nsec): min=6031, '
                                             'max=97032, avg=11358.84, '
                                             'stdev=3221.53\n'
                                             '     lat (nsec): min=6202, '
                                             'max=97222, avg=11533.14, '
                                             'stdev=3199.84\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 6496],  '
                                             '5.00th=[ 7392], 10.00th=[ 8032], '
                                             '20.00th=[ 8512],\n'
                                             '     | 30.00th=[ 8896], '
                                             '40.00th=[ 9664], '
                                             '50.00th=[11712], '
                                             '60.00th=[12608],\n'
                                             '     | 70.00th=[12992], '
                                             '80.00th=[13376], '
                                             '90.00th=[13760], '
                                             '95.00th=[17536],\n'
                                             '     | 99.00th=[21376], '
                                             '99.50th=[23936], '
                                             '99.90th=[29568], '
                                             '99.95th=[35072],\n'
                                             '     | 99.99th=[66048]\n'
                                             '   bw (  KiB/s): min=161805, '
                                             'max=165426, per=100.00%, '
                                             'avg=163730.93, stdev=1280.29, '
                                             'samples=14\n'
                                             '   iops        : min=323611, '
                                             'max=330852, avg=327461.93, '
                                             'stdev=2560.46, samples=14\n'
                                             '  lat (usec)   : 10=41.21%, '
                                             '20=57.59%, 50=1.17%, 100=0.04%\n'
                                             '  cpu          : usr=42.84%, '
                                             'sys=57.13%, ctx=16, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=9.0%, '
                                             '2=45.0%, 4=46.0%, 8=0.0%, '
                                             '16=0.0%, 32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=2291281,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=4\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=160MiB/s (168MB/s), '
                                             '160MiB/s-160MiB/s '
                                             '(168MB/s-168MB/s), io=1119MiB '
                                             '(1173MB), run=7000-7000msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_240.output'),
                                   'state': {'begin': 1666783201.9737847,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_4_io_uring-bdev_uring_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8595907688140869,
                                             'end': 1666783202.8333755,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_240.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_240.state')},
                       'cmd_241': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_512_4_io_uring-bdev_uring_2.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_241.output'),
                                   'state': {'begin': 1666783204.9390643,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_4_io_uring-bdev_uring_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7746074199676514,
                                             'end': 1666783205.7136717,
                                             'err': 1,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_241.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_241.state')},
                       'cmd_242': {'output': 'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_242.output'),
                                   'state': {'begin': 1666783205.7160082,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring-bdev_uring" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="4" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="spdk_bdev" '
                                                    '--spdk_conf="/tmp/spdk_bdev_uring_foo.conf" '
                                                    '--filename="Nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_4_io_uring-bdev_uring_2.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.491530179977417,
                                             'end': 1666783217.2075384,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_242.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_242.state')},
                       'cmd_243': {'output': 'io_uring-bdev_uring: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=spdk_bdev, iodepth=4\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666790307,\n'
                                             '  "timestamp_ms" : '
                                             '1666790307183,\n'
                                             '  "time" : "Wed Oct 26 15:18:27 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring-bdev_uring",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring-bdev_uring",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "4",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"spdk_bdev",\n'
                                             '        "spdk_conf" : '
                                             '"/tmp/spdk_bdev_uring_foo.conf",\n'
                                             '        "filename" : "Nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '1226106368,\n'
                                             '        "io_kbytes" : 1197369,\n'
                                             '        "bw_bytes" : 175133033,\n'
                                             '        "bw" : 171028,\n'
                                             '        "iops" : 342056.277675,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 2394736,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 110,\n'
                                             '          "max" : 17543,\n'
                                             '          "mean" : 182.469767,\n'
                                             '          "stddev" : '
                                             '147.807901,\n'
                                             '          "N" : 2394736\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 5981,\n'
                                             '          "max" : 134212,\n'
                                             '          "mean" : '
                                             '10884.014790,\n'
                                             '          "stddev" : '
                                             '3259.361616,\n'
                                             '          "N" : 2394738,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 6496,\n'
                                             '            "5.000000" : 7392,\n'
                                             '            "10.000000" : 7968,\n'
                                             '            "20.000000" : 8384,\n'
                                             '            "30.000000" : 8768,\n'
                                             '            "40.000000" : 9024,\n'
                                             '            "50.000000" : 9664,\n'
                                             '            "60.000000" : '
                                             '11712,\n'
                                             '            "70.000000" : '
                                             '12608,\n'
                                             '            "80.000000" : '
                                             '13120,\n'
                                             '            "90.000000" : '
                                             '13760,\n'
                                             '            "95.000000" : '
                                             '17536,\n'
                                             '            "99.000000" : '
                                             '21376,\n'
                                             '            "99.500000" : '
                                             '24192,\n'
                                             '            "99.900000" : '
                                             '30080,\n'
                                             '            "99.950000" : '
                                             '35072,\n'
                                             '            "99.990000" : 67072\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 6171,\n'
                                             '          "max" : 134342,\n'
                                             '          "mean" : '
                                             '11066.484526,\n'
                                             '          "stddev" : '
                                             '3234.398550,\n'
                                             '          "N" : 2394738\n'
                                             '        },\n'
                                             '        "bw_min" : 167302,\n'
                                             '        "bw_max" : 174220,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '171056.571429,\n'
                                             '        "bw_dev" : 2202.020531,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 334604,\n'
                                             '        "iops_max" : 348440,\n'
                                             '        "iops_mean" : '
                                             '342113.142857,\n'
                                             '        "iops_stddev" : '
                                             '4404.041061,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 38.157143,\n'
                                             '      "sys_cpu" : 61.814286,\n'
                                             '      "ctx" : 13,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 5.818804,\n'
                                             '        "2" : 42.767136,\n'
                                             '        "4" : 51.414060,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 51.271330,\n'
                                             '        "20" : 47.557309,\n'
                                             '        "50" : 1.134739,\n'
                                             '        "100" : 0.035996,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 4,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring-bdev_uring: (groupid=0, '
                                             'jobs=1): err= 0: pid=9088: Wed '
                                             'Oct 26 15:18:27 2022\n'
                                             '  read: IOPS=342k, BW=167MiB/s '
                                             '(175MB/s)(1169MiB/7001msec)\n'
                                             '    slat (nsec): min=110, '
                                             'max=17543, avg=182.47, '
                                             'stdev=147.81\n'
                                             '    clat (usec): min=5, max=134, '
                                             'avg=10.88, stdev= 3.26\n'
                                             '     lat (usec): min=6, max=134, '
                                             'avg=11.07, stdev= 3.23\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 6496],  '
                                             '5.00th=[ 7392], 10.00th=[ 7968], '
                                             '20.00th=[ 8384],\n'
                                             '     | 30.00th=[ 8768], '
                                             '40.00th=[ 9024], 50.00th=[ '
                                             '9664], 60.00th=[11712],\n'
                                             '     | 70.00th=[12608], '
                                             '80.00th=[13120], '
                                             '90.00th=[13760], '
                                             '95.00th=[17536],\n'
                                             '     | 99.00th=[21376], '
                                             '99.50th=[24192], '
                                             '99.90th=[30080], '
                                             '99.95th=[35072],\n'
                                             '     | 99.99th=[67072]\n'
                                             '   bw (  KiB/s): min=167302, '
                                             'max=174220, per=100.00%, '
                                             'avg=171056.57, stdev=2202.02, '
                                             'samples=14\n'
                                             '   iops        : min=334604, '
                                             'max=348440, avg=342113.14, '
                                             'stdev=4404.04, samples=14\n'
                                             '  lat (usec)   : 10=51.27%, '
                                             '20=47.56%, 50=1.13%, 100=0.04%, '
                                             '250=0.01%\n'
                                             '  cpu          : usr=38.16%, '
                                             'sys=61.81%, ctx=13, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=5.8%, '
                                             '2=42.8%, 4=51.4%, 8=0.0%, '
                                             '16=0.0%, 32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=2394736,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=4\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=167MiB/s (175MB/s), '
                                             '167MiB/s-167MiB/s '
                                             '(175MB/s-175MB/s), io=1169MiB '
                                             '(1226MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_243.output'),
                                   'state': {'begin': 1666783217.2139854,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_4_io_uring-bdev_uring_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8341388702392578,
                                             'end': 1666783218.0481243,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_243.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_243.state')},
                       'cmd_244': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_512_4_io_uring-bdev_xnvme_0.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_244.output'),
                                   'state': {'begin': 1666783220.1547647,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_4_io_uring-bdev_xnvme_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8091392517089844,
                                             'end': 1666783220.963904,
                                             'err': 1,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_244.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_244.state')},
                       'cmd_245': {'output': 'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_245.output'),
                                   'state': {'begin': 1666783220.9659102,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring-bdev_xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="4" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="spdk_bdev" '
                                                    '--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" '
                                                    '--filename="Nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_4_io_uring-bdev_xnvme_0.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.459925413131714,
                                             'end': 1666783232.4258356,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_245.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_245.state')},
                       'cmd_246': {'output': 'io_uring-bdev_xnvme: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=spdk_bdev, iodepth=4\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666790322,\n'
                                             '  "timestamp_ms" : '
                                             '1666790322440,\n'
                                             '  "time" : "Wed Oct 26 15:18:42 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring-bdev_xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring-bdev_xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "4",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"spdk_bdev",\n'
                                             '        "spdk_conf" : '
                                             '"/tmp/spdk_bdev_xnvme_io_uring.conf",\n'
                                             '        "filename" : "Nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '1559368192,\n'
                                             '        "io_kbytes" : 1522820,\n'
                                             '        "bw_bytes" : 222735065,\n'
                                             '        "bw" : 217514,\n'
                                             '        "iops" : 435028.853021,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 3045637,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1603,\n'
                                             '          "max" : 68709,\n'
                                             '          "mean" : 1767.973366,\n'
                                             '          "stddev" : '
                                             '121.396579,\n'
                                             '          "N" : 3045638\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 220,\n'
                                             '          "max" : 119765,\n'
                                             '          "mean" : 7169.615502,\n'
                                             '          "stddev" : '
                                             '2186.549003,\n'
                                             '          "N" : 3045640,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 6624,\n'
                                             '            "5.000000" : 6688,\n'
                                             '            "10.000000" : 6688,\n'
                                             '            "20.000000" : 6688,\n'
                                             '            "30.000000" : 6752,\n'
                                             '            "40.000000" : 6752,\n'
                                             '            "50.000000" : 6752,\n'
                                             '            "60.000000" : 6816,\n'
                                             '            "70.000000" : 6816,\n'
                                             '            "80.000000" : 6880,\n'
                                             '            "90.000000" : 7072,\n'
                                             '            "95.000000" : 7328,\n'
                                             '            "99.000000" : '
                                             '18816,\n'
                                             '            "99.500000" : '
                                             '21888,\n'
                                             '            "99.900000" : '
                                             '25216,\n'
                                             '            "99.950000" : '
                                             '32128,\n'
                                             '            "99.990000" : 66048\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 5411,\n'
                                             '          "max" : 121538,\n'
                                             '          "mean" : 8937.588861,\n'
                                             '          "stddev" : '
                                             '2192.642066,\n'
                                             '          "N" : 3045640\n'
                                             '        },\n'
                                             '        "bw_min" : 216461,\n'
                                             '        "bw_max" : 219449,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '217555.357143,\n'
                                             '        "bw_dev" : 811.925310,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 432922,\n'
                                             '        "iops_max" : 438898,\n'
                                             '        "iops_mean" : '
                                             '435110.714286,\n'
                                             '        "iops_stddev" : '
                                             '1623.850620,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 29.928571,\n'
                                             '      "sys_cpu" : 70.042857,\n'
                                             '      "ctx" : 17,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.010000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 95.836306,\n'
                                             '        "20" : 3.568055,\n'
                                             '        "50" : 0.557814,\n'
                                             '        "100" : 0.036511,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 4,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring-bdev_xnvme: (groupid=0, '
                                             'jobs=1): err= 0: pid=9145: Wed '
                                             'Oct 26 15:18:42 2022\n'
                                             '  read: IOPS=435k, BW=212MiB/s '
                                             '(223MB/s)(1487MiB/7001msec)\n'
                                             '    slat (nsec): min=1603, '
                                             'max=68709, avg=1767.97, '
                                             'stdev=121.40\n'
                                             '    clat (nsec): min=220, '
                                             'max=119765, avg=7169.62, '
                                             'stdev=2186.55\n'
                                             '     lat (usec): min=5, max=121, '
                                             'avg= 8.94, stdev= 2.19\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 6624],  '
                                             '5.00th=[ 6688], 10.00th=[ 6688], '
                                             '20.00th=[ 6688],\n'
                                             '     | 30.00th=[ 6752], '
                                             '40.00th=[ 6752], 50.00th=[ '
                                             '6752], 60.00th=[ 6816],\n'
                                             '     | 70.00th=[ 6816], '
                                             '80.00th=[ 6880], 90.00th=[ '
                                             '7072], 95.00th=[ 7328],\n'
                                             '     | 99.00th=[18816], '
                                             '99.50th=[21888], '
                                             '99.90th=[25216], '
                                             '99.95th=[32128],\n'
                                             '     | 99.99th=[66048]\n'
                                             '   bw (  KiB/s): min=216461, '
                                             'max=219449, per=100.00%, '
                                             'avg=217555.36, stdev=811.93, '
                                             'samples=14\n'
                                             '   iops        : min=432922, '
                                             'max=438898, avg=435110.71, '
                                             'stdev=1623.85, samples=14\n'
                                             '  lat (nsec)   : 250=0.01%, '
                                             '500=0.01%\n'
                                             '  lat (usec)   : 4=0.01%, '
                                             '10=95.84%, 20=3.57%, 50=0.56%, '
                                             '100=0.04%\n'
                                             '  lat (usec)   : 250=0.01%\n'
                                             '  cpu          : usr=29.93%, '
                                             'sys=70.04%, ctx=17, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, 2=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=3045637,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=4\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=212MiB/s (223MB/s), '
                                             '212MiB/s-212MiB/s '
                                             '(223MB/s-223MB/s), io=1487MiB '
                                             '(1559MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_246.output'),
                                   'state': {'begin': 1666783232.428395,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_4_io_uring-bdev_xnvme_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.80635666847229,
                                             'end': 1666783233.2347517,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_246.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_246.state')},
                       'cmd_247': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_512_4_io_uring-bdev_xnvme_1.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_247.output'),
                                   'state': {'begin': 1666783235.2788057,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_4_io_uring-bdev_xnvme_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8425042629241943,
                                             'end': 1666783236.12131,
                                             'err': 1,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_247.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_247.state')},
                       'cmd_248': {'output': 'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_248.output'),
                                   'state': {'begin': 1666783236.1232603,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring-bdev_xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="4" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="spdk_bdev" '
                                                    '--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" '
                                                    '--filename="Nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_4_io_uring-bdev_xnvme_1.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.452204465866089,
                                             'end': 1666783247.5754647,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_248.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_248.state')},
                       'cmd_249': {'output': 'io_uring-bdev_xnvme: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=spdk_bdev, iodepth=4\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666790337,\n'
                                             '  "timestamp_ms" : '
                                             '1666790337589,\n'
                                             '  "time" : "Wed Oct 26 15:18:57 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring-bdev_xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring-bdev_xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "4",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"spdk_bdev",\n'
                                             '        "spdk_conf" : '
                                             '"/tmp/spdk_bdev_xnvme_io_uring.conf",\n'
                                             '        "filename" : "Nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '1562582528,\n'
                                             '        "io_kbytes" : 1525959,\n'
                                             '        "bw_bytes" : 223194190,\n'
                                             '        "bw" : 217963,\n'
                                             '        "iops" : 435925.582060,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 3051915,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1593,\n'
                                             '          "max" : 23705,\n'
                                             '          "mean" : 1775.126412,\n'
                                             '          "stddev" : '
                                             '116.995816,\n'
                                             '          "N" : 3051916\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 231,\n'
                                             '          "max" : 117450,\n'
                                             '          "mean" : 7161.374734,\n'
                                             '          "stddev" : '
                                             '2248.409210,\n'
                                             '          "N" : 3051918,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 6624,\n'
                                             '            "5.000000" : 6624,\n'
                                             '            "10.000000" : 6688,\n'
                                             '            "20.000000" : 6688,\n'
                                             '            "30.000000" : 6752,\n'
                                             '            "40.000000" : 6752,\n'
                                             '            "50.000000" : 6752,\n'
                                             '            "60.000000" : 6752,\n'
                                             '            "70.000000" : 6816,\n'
                                             '            "80.000000" : 6880,\n'
                                             '            "90.000000" : 7072,\n'
                                             '            "95.000000" : 7264,\n'
                                             '            "99.000000" : '
                                             '19328,\n'
                                             '            "99.500000" : '
                                             '24192,\n'
                                             '            "99.900000" : '
                                             '25728,\n'
                                             '            "99.950000" : '
                                             '32128,\n'
                                             '            "99.990000" : 66048\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 5370,\n'
                                             '          "max" : 119194,\n'
                                             '          "mean" : 8936.501126,\n'
                                             '          "stddev" : '
                                             '2254.690146,\n'
                                             '          "N" : 3051918\n'
                                             '        },\n'
                                             '        "bw_min" : 216970,\n'
                                             '        "bw_max" : 219040,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '218076.500000,\n'
                                             '        "bw_dev" : 597.526279,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 433940,\n'
                                             '        "iops_max" : 438080,\n'
                                             '        "iops_mean" : '
                                             '436153.071429,\n'
                                             '        "iops_stddev" : '
                                             '1194.975537,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 30.000000,\n'
                                             '      "sys_cpu" : 69.985714,\n'
                                             '      "ctx" : 10,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.010000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 96.118994,\n'
                                             '        "20" : 3.222043,\n'
                                             '        "50" : 0.619414,\n'
                                             '        "100" : 0.035486,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 4,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring-bdev_xnvme: (groupid=0, '
                                             'jobs=1): err= 0: pid=9203: Wed '
                                             'Oct 26 15:18:57 2022\n'
                                             '  read: IOPS=436k, BW=213MiB/s '
                                             '(223MB/s)(1490MiB/7001msec)\n'
                                             '    slat (nsec): min=1593, '
                                             'max=23705, avg=1775.13, '
                                             'stdev=117.00\n'
                                             '    clat (nsec): min=231, '
                                             'max=117450, avg=7161.37, '
                                             'stdev=2248.41\n'
                                             '     lat (usec): min=5, max=119, '
                                             'avg= 8.94, stdev= 2.25\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 6624],  '
                                             '5.00th=[ 6624], 10.00th=[ 6688], '
                                             '20.00th=[ 6688],\n'
                                             '     | 30.00th=[ 6752], '
                                             '40.00th=[ 6752], 50.00th=[ '
                                             '6752], 60.00th=[ 6752],\n'
                                             '     | 70.00th=[ 6816], '
                                             '80.00th=[ 6880], 90.00th=[ '
                                             '7072], 95.00th=[ 7264],\n'
                                             '     | 99.00th=[19328], '
                                             '99.50th=[24192], '
                                             '99.90th=[25728], '
                                             '99.95th=[32128],\n'
                                             '     | 99.99th=[66048]\n'
                                             '   bw (  KiB/s): min=216970, '
                                             'max=219040, per=100.00%, '
                                             'avg=218076.50, stdev=597.53, '
                                             'samples=14\n'
                                             '   iops        : min=433940, '
                                             'max=438080, avg=436153.07, '
                                             'stdev=1194.98, samples=14\n'
                                             '  lat (nsec)   : 250=0.01%, '
                                             '500=0.01%\n'
                                             '  lat (usec)   : 4=0.01%, '
                                             '10=96.12%, 20=3.22%, 50=0.62%, '
                                             '100=0.04%\n'
                                             '  lat (usec)   : 250=0.01%\n'
                                             '  cpu          : usr=30.00%, '
                                             'sys=69.99%, ctx=10, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, 2=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=3051915,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=4\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=213MiB/s (223MB/s), '
                                             '213MiB/s-213MiB/s '
                                             '(223MB/s-223MB/s), io=1490MiB '
                                             '(1563MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_249.output'),
                                   'state': {'begin': 1666783247.578205,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_4_io_uring-bdev_xnvme_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8199121952056885,
                                             'end': 1666783248.3981173,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_249.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_249.state')},
                       'cmd_25': {'output': 'rm: cannot remove '
                                            "'/tmp/fio-output_512_1_io_uring_cmd-bdev_xnvme_2.txt': "
                                            'No such file or directory\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_25.output'),
                                  'state': {'begin': 1666782161.0840642,
                                            'cmd': 'rm '
                                                   '/tmp/fio-output_512_1_io_uring_cmd-bdev_xnvme_2.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.8233551979064941,
                                            'end': 1666782161.9074194,
                                            'err': 1,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_25.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_25.state')},
                       'cmd_250': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_512_4_io_uring-bdev_xnvme_2.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_250.output'),
                                   'state': {'begin': 1666783250.4902697,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_4_io_uring-bdev_xnvme_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.801767110824585,
                                             'end': 1666783251.2920368,
                                             'err': 1,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_250.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_250.state')},
                       'cmd_251': {'output': 'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_251.output'),
                                   'state': {'begin': 1666783251.293977,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring-bdev_xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="4" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="spdk_bdev" '
                                                    '--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" '
                                                    '--filename="Nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_4_io_uring-bdev_xnvme_2.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.309634685516357,
                                             'end': 1666783262.6036117,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_251.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_251.state')},
                       'cmd_252': {'output': 'io_uring-bdev_xnvme: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=spdk_bdev, iodepth=4\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666790352,\n'
                                             '  "timestamp_ms" : '
                                             '1666790352617,\n'
                                             '  "time" : "Wed Oct 26 15:19:12 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring-bdev_xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring-bdev_xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "4",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"spdk_bdev",\n'
                                             '        "spdk_conf" : '
                                             '"/tmp/spdk_bdev_xnvme_io_uring.conf",\n'
                                             '        "filename" : "Nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '1559137792,\n'
                                             '        "io_kbytes" : 1522595,\n'
                                             '        "bw_bytes" : 222702155,\n'
                                             '        "bw" : 217482,\n'
                                             '        "iops" : 434964.576489,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 3045187,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1593,\n'
                                             '          "max" : 20638,\n'
                                             '          "mean" : 1801.881652,\n'
                                             '          "stddev" : '
                                             '112.241295,\n'
                                             '          "N" : 3045188\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 211,\n'
                                             '          "max" : 117270,\n'
                                             '          "mean" : 7169.162602,\n'
                                             '          "stddev" : '
                                             '2248.200291,\n'
                                             '          "N" : 3045190,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 6624,\n'
                                             '            "5.000000" : 6624,\n'
                                             '            "10.000000" : 6688,\n'
                                             '            "20.000000" : 6688,\n'
                                             '            "30.000000" : 6752,\n'
                                             '            "40.000000" : 6752,\n'
                                             '            "50.000000" : 6752,\n'
                                             '            "60.000000" : 6816,\n'
                                             '            "70.000000" : 6816,\n'
                                             '            "80.000000" : 6880,\n'
                                             '            "90.000000" : 7072,\n'
                                             '            "95.000000" : 7264,\n'
                                             '            "99.000000" : '
                                             '19328,\n'
                                             '            "99.500000" : '
                                             '24192,\n'
                                             '            "99.900000" : '
                                             '25728,\n'
                                             '            "99.950000" : '
                                             '32384,\n'
                                             '            "99.990000" : 67072\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 5360,\n'
                                             '          "max" : 118993,\n'
                                             '          "mean" : 8971.044281,\n'
                                             '          "stddev" : '
                                             '2253.640954,\n'
                                             '          "N" : 3045190\n'
                                             '        },\n'
                                             '        "bw_min" : 216412,\n'
                                             '        "bw_max" : 218927,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '217521.785714,\n'
                                             '        "bw_dev" : 661.670750,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 432824,\n'
                                             '        "iops_max" : 437854,\n'
                                             '        "iops_mean" : '
                                             '435043.642857,\n'
                                             '        "iops_stddev" : '
                                             '1323.496628,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 32.385714,\n'
                                             '      "sys_cpu" : 67.600000,\n'
                                             '      "ctx" : 14,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.010000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 96.063559,\n'
                                             '        "20" : 3.301308,\n'
                                             '        "50" : 0.594709,\n'
                                             '        "100" : 0.036911,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 4,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring-bdev_xnvme: (groupid=0, '
                                             'jobs=1): err= 0: pid=9260: Wed '
                                             'Oct 26 15:19:12 2022\n'
                                             '  read: IOPS=435k, BW=212MiB/s '
                                             '(223MB/s)(1487MiB/7001msec)\n'
                                             '    slat (nsec): min=1593, '
                                             'max=20638, avg=1801.88, '
                                             'stdev=112.24\n'
                                             '    clat (nsec): min=211, '
                                             'max=117270, avg=7169.16, '
                                             'stdev=2248.20\n'
                                             '     lat (usec): min=5, max=118, '
                                             'avg= 8.97, stdev= 2.25\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 6624],  '
                                             '5.00th=[ 6624], 10.00th=[ 6688], '
                                             '20.00th=[ 6688],\n'
                                             '     | 30.00th=[ 6752], '
                                             '40.00th=[ 6752], 50.00th=[ '
                                             '6752], 60.00th=[ 6816],\n'
                                             '     | 70.00th=[ 6816], '
                                             '80.00th=[ 6880], 90.00th=[ '
                                             '7072], 95.00th=[ 7264],\n'
                                             '     | 99.00th=[19328], '
                                             '99.50th=[24192], '
                                             '99.90th=[25728], '
                                             '99.95th=[32384],\n'
                                             '     | 99.99th=[67072]\n'
                                             '   bw (  KiB/s): min=216412, '
                                             'max=218927, per=100.00%, '
                                             'avg=217521.79, stdev=661.67, '
                                             'samples=14\n'
                                             '   iops        : min=432824, '
                                             'max=437854, avg=435043.64, '
                                             'stdev=1323.50, samples=14\n'
                                             '  lat (nsec)   : 250=0.01%, '
                                             '500=0.01%\n'
                                             '  lat (usec)   : 4=0.01%, '
                                             '10=96.06%, 20=3.30%, 50=0.59%, '
                                             '100=0.04%\n'
                                             '  lat (usec)   : 250=0.01%\n'
                                             '  cpu          : usr=32.39%, '
                                             'sys=67.60%, ctx=14, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, 2=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=3045187,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=4\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=212MiB/s (223MB/s), '
                                             '212MiB/s-212MiB/s '
                                             '(223MB/s-223MB/s), io=1487MiB '
                                             '(1559MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_252.output'),
                                   'state': {'begin': 1666783262.6063745,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_4_io_uring-bdev_xnvme_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8146522045135498,
                                             'end': 1666783263.4210267,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_252.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_252.state')},
                       'cmd_253': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_512_4_io_uring-xnvme_0.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_253.output'),
                                   'state': {'begin': 1666783264.4408336,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_4_io_uring-xnvme_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7997441291809082,
                                             'end': 1666783265.2405777,
                                             'err': 1,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_253.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_253.state')},
                       'cmd_254': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_254.output'),
                                   'state': {'begin': 1666783265.242522,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring-xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="4" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
                                                    '--xnvme_be="linux" '
                                                    '--xnvme_async="io_uring" '
                                                    '--xnvme_sync="nvme" '
                                                    '--xnvme_admin="nvme" '
                                                    '--xnvme_dev_nsid="1" '
                                                    '--filename="/dev/nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_4_io_uring-xnvme_0.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.078611850738525,
                                             'end': 1666783276.3211339,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_254.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_254.state')},
                       'cmd_255': {'output': 'io_uring-xnvme: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=xnvme, iodepth=4\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666790366,\n'
                                             '  "timestamp_ms" : '
                                             '1666790366377,\n'
                                             '  "time" : "Wed Oct 26 15:19:26 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring-xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring-xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "4",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
                                             '        "xnvme_be" : "linux",\n'
                                             '        "xnvme_async" : '
                                             '"io_uring",\n'
                                             '        "xnvme_sync" : "nvme",\n'
                                             '        "xnvme_admin" : "nvme",\n'
                                             '        "xnvme_dev_nsid" : "1",\n'
                                             '        "filename" : '
                                             '"/dev/nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '1420984832,\n'
                                             '        "io_kbytes" : 1387680,\n'
                                             '        "bw_bytes" : 202968837,\n'
                                             '        "bw" : 198211,\n'
                                             '        "iops" : 396423.082417,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 2775358,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1382,\n'
                                             '          "max" : 67266,\n'
                                             '          "mean" : 1980.791384,\n'
                                             '          "stddev" : '
                                             '330.353282,\n'
                                             '          "N" : 2775358\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 2054,\n'
                                             '          "max" : 124162,\n'
                                             '          "mean" : 7810.983699,\n'
                                             '          "stddev" : '
                                             '2690.900419,\n'
                                             '          "N" : 2775361,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 4448,\n'
                                             '            "5.000000" : 5152,\n'
                                             '            "10.000000" : 6816,\n'
                                             '            "20.000000" : 6944,\n'
                                             '            "30.000000" : 7392,\n'
                                             '            "40.000000" : 7456,\n'
                                             '            "50.000000" : 7456,\n'
                                             '            "60.000000" : 7520,\n'
                                             '            "70.000000" : 7648,\n'
                                             '            "80.000000" : 8032,\n'
                                             '            "90.000000" : 8384,\n'
                                             '            "95.000000" : 9408,\n'
                                             '            "99.000000" : '
                                             '20864,\n'
                                             '            "99.500000" : '
                                             '25216,\n'
                                             '            "99.900000" : '
                                             '27264,\n'
                                             '            "99.950000" : '
                                             '33536,\n'
                                             '            "99.990000" : 69120\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 5781,\n'
                                             '          "max" : 126286,\n'
                                             '          "mean" : 9791.775021,\n'
                                             '          "stddev" : '
                                             '2666.740323,\n'
                                             '          "N" : 2775361\n'
                                             '        },\n'
                                             '        "bw_min" : 197094,\n'
                                             '        "bw_max" : 199345,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '198313.214286,\n'
                                             '        "bw_dev" : 631.854801,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 394188,\n'
                                             '        "iops_max" : 398690,\n'
                                             '        "iops_mean" : '
                                             '396626.571429,\n'
                                             '        "iops_stddev" : '
                                             '1263.775583,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 17.671429,\n'
                                             '      "sys_cpu" : 78.628571,\n'
                                             '      "ctx" : 27529,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 96.050924,\n'
                                             '        "20" : 2.666179,\n'
                                             '        "50" : 1.239480,\n'
                                             '        "100" : 0.041184,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 4,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring-xnvme: (groupid=0, '
                                             'jobs=1): err= 0: pid=9307: Wed '
                                             'Oct 26 15:19:26 2022\n'
                                             '  read: IOPS=396k, BW=194MiB/s '
                                             '(203MB/s)(1355MiB/7001msec)\n'
                                             '    slat (nsec): min=1382, '
                                             'max=67266, avg=1980.79, '
                                             'stdev=330.35\n'
                                             '    clat (usec): min=2, max=124, '
                                             'avg= 7.81, stdev= 2.69\n'
                                             '     lat (usec): min=5, max=126, '
                                             'avg= 9.79, stdev= 2.67\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 4448],  '
                                             '5.00th=[ 5152], 10.00th=[ 6816], '
                                             '20.00th=[ 6944],\n'
                                             '     | 30.00th=[ 7392], '
                                             '40.00th=[ 7456], 50.00th=[ '
                                             '7456], 60.00th=[ 7520],\n'
                                             '     | 70.00th=[ 7648], '
                                             '80.00th=[ 8032], 90.00th=[ '
                                             '8384], 95.00th=[ 9408],\n'
                                             '     | 99.00th=[20864], '
                                             '99.50th=[25216], '
                                             '99.90th=[27264], '
                                             '99.95th=[33536],\n'
                                             '     | 99.99th=[69120]\n'
                                             '   bw (  KiB/s): min=197094, '
                                             'max=199345, per=100.00%, '
                                             'avg=198313.21, stdev=631.85, '
                                             'samples=14\n'
                                             '   iops        : min=394188, '
                                             'max=398690, avg=396626.57, '
                                             'stdev=1263.78, samples=14\n'
                                             '  lat (usec)   : 4=0.01%, '
                                             '10=96.05%, 20=2.67%, 50=1.24%, '
                                             '100=0.04%\n'
                                             '  lat (usec)   : 250=0.01%\n'
                                             '  cpu          : usr=17.67%, '
                                             'sys=78.63%, ctx=27529, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, 2=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=2775358,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=4\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=194MiB/s (203MB/s), '
                                             '194MiB/s-194MiB/s '
                                             '(203MB/s-203MB/s), io=1355MiB '
                                             '(1421MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_255.output'),
                                   'state': {'begin': 1666783276.3240511,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_4_io_uring-xnvme_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8232474327087402,
                                             'end': 1666783277.1472986,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_255.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_255.state')},
                       'cmd_256': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_512_4_io_uring-xnvme_1.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_256.output'),
                                   'state': {'begin': 1666783278.1612356,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_4_io_uring-xnvme_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8103094100952148,
                                             'end': 1666783278.971545,
                                             'err': 1,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_256.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_256.state')},
                       'cmd_257': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_257.output'),
                                   'state': {'begin': 1666783278.9736066,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring-xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="4" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
                                                    '--xnvme_be="linux" '
                                                    '--xnvme_async="io_uring" '
                                                    '--xnvme_sync="nvme" '
                                                    '--xnvme_admin="nvme" '
                                                    '--xnvme_dev_nsid="1" '
                                                    '--filename="/dev/nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_4_io_uring-xnvme_1.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.077608346939087,
                                             'end': 1666783290.051215,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_257.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_257.state')},
                       'cmd_258': {'output': 'io_uring-xnvme: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=xnvme, iodepth=4\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666790380,\n'
                                             '  "timestamp_ms" : '
                                             '1666790380127,\n'
                                             '  "time" : "Wed Oct 26 15:19:40 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring-xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring-xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "4",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
                                             '        "xnvme_be" : "linux",\n'
                                             '        "xnvme_async" : '
                                             '"io_uring",\n'
                                             '        "xnvme_sync" : "nvme",\n'
                                             '        "xnvme_admin" : "nvme",\n'
                                             '        "xnvme_dev_nsid" : "1",\n'
                                             '        "filename" : '
                                             '"/dev/nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '1402958336,\n'
                                             '        "io_kbytes" : 1370076,\n'
                                             '        "bw_bytes" : 200393991,\n'
                                             '        "bw" : 195697,\n'
                                             '        "iops" : 391393.943722,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 2740149,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1383,\n'
                                             '          "max" : 22943,\n'
                                             '          "mean" : 2020.769172,\n'
                                             '          "stddev" : '
                                             '287.667604,\n'
                                             '          "N" : 2740150\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 2103,\n'
                                             '          "max" : 122420,\n'
                                             '          "mean" : 7908.474523,\n'
                                             '          "stddev" : '
                                             '2587.271863,\n'
                                             '          "N" : 2740152,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 4512,\n'
                                             '            "5.000000" : 6816,\n'
                                             '            "10.000000" : 6880,\n'
                                             '            "20.000000" : 7456,\n'
                                             '            "30.000000" : 7456,\n'
                                             '            "40.000000" : 7520,\n'
                                             '            "50.000000" : 7520,\n'
                                             '            "60.000000" : 7584,\n'
                                             '            "70.000000" : 7648,\n'
                                             '            "80.000000" : 7840,\n'
                                             '            "90.000000" : 8256,\n'
                                             '            "95.000000" : 9024,\n'
                                             '            "99.000000" : '
                                             '20608,\n'
                                             '            "99.500000" : '
                                             '24960,\n'
                                             '            "99.900000" : '
                                             '27008,\n'
                                             '            "99.950000" : '
                                             '33024,\n'
                                             '            "99.990000" : 69120\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 5590,\n'
                                             '          "max" : 123922,\n'
                                             '          "mean" : 9929.243570,\n'
                                             '          "stddev" : '
                                             '2555.049304,\n'
                                             '          "N" : 2740152\n'
                                             '        },\n'
                                             '        "bw_min" : 194510,\n'
                                             '        "bw_max" : 197872,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '195731.571429,\n'
                                             '        "bw_dev" : 1047.071059,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 389020,\n'
                                             '        "iops_max" : 395744,\n'
                                             '        "iops_mean" : '
                                             '391463.214286,\n'
                                             '        "iops_stddev" : '
                                             '2094.197044,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 20.785714,\n'
                                             '      "sys_cpu" : 75.628571,\n'
                                             '      "ctx" : 27189,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 96.072622,\n'
                                             '        "20" : 2.739012,\n'
                                             '        "50" : 1.149791,\n'
                                             '        "100" : 0.037115,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 4,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring-xnvme: (groupid=0, '
                                             'jobs=1): err= 0: pid=9355: Wed '
                                             'Oct 26 15:19:40 2022\n'
                                             '  read: IOPS=391k, BW=191MiB/s '
                                             '(200MB/s)(1338MiB/7001msec)\n'
                                             '    slat (nsec): min=1383, '
                                             'max=22943, avg=2020.77, '
                                             'stdev=287.67\n'
                                             '    clat (usec): min=2, max=122, '
                                             'avg= 7.91, stdev= 2.59\n'
                                             '     lat (usec): min=5, max=123, '
                                             'avg= 9.93, stdev= 2.56\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 4512],  '
                                             '5.00th=[ 6816], 10.00th=[ 6880], '
                                             '20.00th=[ 7456],\n'
                                             '     | 30.00th=[ 7456], '
                                             '40.00th=[ 7520], 50.00th=[ '
                                             '7520], 60.00th=[ 7584],\n'
                                             '     | 70.00th=[ 7648], '
                                             '80.00th=[ 7840], 90.00th=[ '
                                             '8256], 95.00th=[ 9024],\n'
                                             '     | 99.00th=[20608], '
                                             '99.50th=[24960], '
                                             '99.90th=[27008], '
                                             '99.95th=[33024],\n'
                                             '     | 99.99th=[69120]\n'
                                             '   bw (  KiB/s): min=194510, '
                                             'max=197872, per=100.00%, '
                                             'avg=195731.57, stdev=1047.07, '
                                             'samples=14\n'
                                             '   iops        : min=389020, '
                                             'max=395744, avg=391463.21, '
                                             'stdev=2094.20, samples=14\n'
                                             '  lat (usec)   : 4=0.01%, '
                                             '10=96.07%, 20=2.74%, 50=1.15%, '
                                             '100=0.04%\n'
                                             '  lat (usec)   : 250=0.01%\n'
                                             '  cpu          : usr=20.79%, '
                                             'sys=75.63%, ctx=27189, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, 2=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=2740149,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=4\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=191MiB/s (200MB/s), '
                                             '191MiB/s-191MiB/s '
                                             '(200MB/s-200MB/s), io=1338MiB '
                                             '(1403MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_258.output'),
                                   'state': {'begin': 1666783290.054493,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_4_io_uring-xnvme_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8424615859985352,
                                             'end': 1666783290.8969545,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_258.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_258.state')},
                       'cmd_259': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_512_4_io_uring-xnvme_2.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_259.output'),
                                   'state': {'begin': 1666783291.9385934,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_4_io_uring-xnvme_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8601806163787842,
                                             'end': 1666783292.798774,
                                             'err': 1,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_259.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_259.state')},
                       'cmd_26': {'output': 'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_26.output'),
                                  'state': {'begin': 1666782161.9096975,
                                            'cmd': '/opt/fio/bin/fio '
                                                   '--name="io_uring_cmd-bdev_xnvme" '
                                                   '--rw="randread" '
                                                   '--size="12G" --iodepth="1" '
                                                   '--bs="512" --direct="0" '
                                                   '--thread="1" '
                                                   '--time_based="1" '
                                                   '--runtime="7" '
                                                   '--ramp_time="3" '
                                                   '--norandommap="1" '
                                                   '--allow_file_create="0" '
                                                   '--ioengine="spdk_bdev" '
                                                   '--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" '
                                                   '--filename="Nvme0n1" '
                                                   '--output-format="normal,json" '
                                                   '--output="/tmp/fio-output_512_1_io_uring_cmd-bdev_xnvme_2.txt" '
                                                   '--cpus_allowed="1"',
                                            'cwd': 'None',
                                            'elapsed': 11.442041873931885,
                                            'end': 1666782173.3517394,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_26.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_26.state')},
                       'cmd_260': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_260.output'),
                                   'state': {'begin': 1666783292.8010087,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring-xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="4" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
                                                    '--xnvme_be="linux" '
                                                    '--xnvme_async="io_uring" '
                                                    '--xnvme_sync="nvme" '
                                                    '--xnvme_admin="nvme" '
                                                    '--xnvme_dev_nsid="1" '
                                                    '--filename="/dev/nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_4_io_uring-xnvme_2.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.117149829864502,
                                             'end': 1666783303.9181585,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_260.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_260.state')},
                       'cmd_261': {'output': 'io_uring-xnvme: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=xnvme, iodepth=4\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666790393,\n'
                                             '  "timestamp_ms" : '
                                             '1666790393982,\n'
                                             '  "time" : "Wed Oct 26 15:19:53 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring-xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring-xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "4",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
                                             '        "xnvme_be" : "linux",\n'
                                             '        "xnvme_async" : '
                                             '"io_uring",\n'
                                             '        "xnvme_sync" : "nvme",\n'
                                             '        "xnvme_admin" : "nvme",\n'
                                             '        "xnvme_dev_nsid" : "1",\n'
                                             '        "filename" : '
                                             '"/dev/nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '1408428544,\n'
                                             '        "io_kbytes" : 1375418,\n'
                                             '        "bw_bytes" : 201175338,\n'
                                             '        "bw" : 196460,\n'
                                             '        "iops" : 392920.011427,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 2750833,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1402,\n'
                                             '          "max" : 43572,\n'
                                             '          "mean" : 2012.219970,\n'
                                             '          "stddev" : '
                                             '307.976683,\n'
                                             '          "N" : 2750834\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 131,\n'
                                             '          "max" : 104275,\n'
                                             '          "mean" : 7874.613087,\n'
                                             '          "stddev" : '
                                             '2621.702740,\n'
                                             '          "N" : 2750836,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 4448,\n'
                                             '            "5.000000" : 6304,\n'
                                             '            "10.000000" : 6880,\n'
                                             '            "20.000000" : 7200,\n'
                                             '            "30.000000" : 7456,\n'
                                             '            "40.000000" : 7520,\n'
                                             '            "50.000000" : 7520,\n'
                                             '            "60.000000" : 7584,\n'
                                             '            "70.000000" : 7648,\n'
                                             '            "80.000000" : 7904,\n'
                                             '            "90.000000" : 8256,\n'
                                             '            "95.000000" : 9280,\n'
                                             '            "99.000000" : '
                                             '20864,\n'
                                             '            "99.500000" : '
                                             '25216,\n'
                                             '            "99.900000" : '
                                             '27008,\n'
                                             '            "99.950000" : '
                                             '33024,\n'
                                             '            "99.990000" : 68096\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 5771,\n'
                                             '          "max" : 106169,\n'
                                             '          "mean" : 9886.833346,\n'
                                             '          "stddev" : '
                                             '2594.294429,\n'
                                             '          "N" : 2750836\n'
                                             '        },\n'
                                             '        "bw_min" : 195388,\n'
                                             '        "bw_max" : 198951,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '196495.071429,\n'
                                             '        "bw_dev" : 1111.164217,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 390776,\n'
                                             '        "iops_max" : 397903,\n'
                                             '        "iops_mean" : '
                                             '392990.214286,\n'
                                             '        "iops_stddev" : '
                                             '2222.498462,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 21.085714,\n'
                                             '      "sys_cpu" : 75.314286,\n'
                                             '      "ctx" : 27181,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 96.066900,\n'
                                             '        "20" : 2.703181,\n'
                                             '        "50" : 1.190040,\n'
                                             '        "100" : 0.038025,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 4,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring-xnvme: (groupid=0, '
                                             'jobs=1): err= 0: pid=9402: Wed '
                                             'Oct 26 15:19:53 2022\n'
                                             '  read: IOPS=393k, BW=192MiB/s '
                                             '(201MB/s)(1343MiB/7001msec)\n'
                                             '    slat (nsec): min=1402, '
                                             'max=43572, avg=2012.22, '
                                             'stdev=307.98\n'
                                             '    clat (nsec): min=131, '
                                             'max=104275, avg=7874.61, '
                                             'stdev=2621.70\n'
                                             '     lat (usec): min=5, max=106, '
                                             'avg= 9.89, stdev= 2.59\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 4448],  '
                                             '5.00th=[ 6304], 10.00th=[ 6880], '
                                             '20.00th=[ 7200],\n'
                                             '     | 30.00th=[ 7456], '
                                             '40.00th=[ 7520], 50.00th=[ '
                                             '7520], 60.00th=[ 7584],\n'
                                             '     | 70.00th=[ 7648], '
                                             '80.00th=[ 7904], 90.00th=[ '
                                             '8256], 95.00th=[ 9280],\n'
                                             '     | 99.00th=[20864], '
                                             '99.50th=[25216], '
                                             '99.90th=[27008], '
                                             '99.95th=[33024],\n'
                                             '     | 99.99th=[68096]\n'
                                             '   bw (  KiB/s): min=195388, '
                                             'max=198951, per=100.00%, '
                                             'avg=196495.07, stdev=1111.16, '
                                             'samples=14\n'
                                             '   iops        : min=390776, '
                                             'max=397903, avg=392990.21, '
                                             'stdev=2222.50, samples=14\n'
                                             '  lat (nsec)   : 250=0.01%\n'
                                             '  lat (usec)   : 4=0.01%, '
                                             '10=96.07%, 20=2.70%, 50=1.19%, '
                                             '100=0.04%\n'
                                             '  lat (usec)   : 250=0.01%\n'
                                             '  cpu          : usr=21.09%, '
                                             'sys=75.31%, ctx=27181, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, 2=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=2750833,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=4\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=192MiB/s (201MB/s), '
                                             '192MiB/s-192MiB/s '
                                             '(201MB/s-201MB/s), io=1343MiB '
                                             '(1408MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_261.output'),
                                   'state': {'begin': 1666783303.9249487,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_4_io_uring-xnvme_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7905657291412354,
                                             'end': 1666783304.7155144,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_261.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_261.state')},
                       'cmd_262': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_512_4_libaio-reference_0.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_262.output'),
                                   'state': {'begin': 1666783305.77147,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_4_libaio-reference_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7845261096954346,
                                             'end': 1666783306.5559962,
                                             'err': 1,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_262.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_262.state')},
                       'cmd_263': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_263.output'),
                                   'state': {'begin': 1666783306.5581977,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="libaio-reference" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="4" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="libaio" '
                                                    '--filename="/dev/nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_4_libaio-reference_0.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.11328387260437,
                                             'end': 1666783317.6714816,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_263.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_263.state')},
                       'cmd_264': {'output': 'libaio-reference: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=libaio, iodepth=4\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666790407,\n'
                                             '  "timestamp_ms" : '
                                             '1666790407746,\n'
                                             '  "time" : "Wed Oct 26 15:20:07 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"libaio-reference",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"libaio-reference",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "4",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : "libaio",\n'
                                             '        "filename" : '
                                             '"/dev/nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '1540268544,\n'
                                             '        "io_kbytes" : 1504168,\n'
                                             '        "bw_bytes" : 220006933,\n'
                                             '        "bw" : 214850,\n'
                                             '        "iops" : 429700.471361,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 3008333,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1442,\n'
                                             '          "max" : 22633,\n'
                                             '          "mean" : 1606.975883,\n'
                                             '          "stddev" : '
                                             '117.237640,\n'
                                             '          "N" : 3008334\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 2945,\n'
                                             '          "max" : 110618,\n'
                                             '          "mean" : 7468.559953,\n'
                                             '          "stddev" : '
                                             '2665.426037,\n'
                                             '          "N" : 3008336,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 6816,\n'
                                             '            "5.000000" : 6880,\n'
                                             '            "10.000000" : 6880,\n'
                                             '            "20.000000" : 6944,\n'
                                             '            "30.000000" : 6944,\n'
                                             '            "40.000000" : 6944,\n'
                                             '            "50.000000" : 7008,\n'
                                             '            "60.000000" : 7008,\n'
                                             '            "70.000000" : 7072,\n'
                                             '            "80.000000" : 7200,\n'
                                             '            "90.000000" : 7264,\n'
                                             '            "95.000000" : 7392,\n'
                                             '            "99.000000" : '
                                             '22400,\n'
                                             '            "99.500000" : '
                                             '27008,\n'
                                             '            "99.900000" : '
                                             '32640,\n'
                                             '            "99.950000" : '
                                             '35072,\n'
                                             '            "99.990000" : 69120\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 6051,\n'
                                             '          "max" : 112181,\n'
                                             '          "mean" : 9075.535866,\n'
                                             '          "stddev" : '
                                             '2669.379295,\n'
                                             '          "N" : 3008336\n'
                                             '        },\n'
                                             '        "bw_min" : 213880,\n'
                                             '        "bw_max" : 215909,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '214911.214286,\n'
                                             '        "bw_dev" : 609.982620,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 427760,\n'
                                             '        "iops_max" : 431818,\n'
                                             '        "iops_mean" : '
                                             '429822.428571,\n'
                                             '        "iops_stddev" : '
                                             '1219.965240,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 28.971429,\n'
                                             '      "sys_cpu" : 67.200000,\n'
                                             '      "ctx" : 26583,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 96.583922,\n'
                                             '        "20" : 2.229574,\n'
                                             '        "50" : 1.152033,\n'
                                             '        "100" : 0.034438,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 4,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ],\n'
                                             '  "disk_util" : [\n'
                                             '    {\n'
                                             '      "name" : "nvme0n1",\n'
                                             '      "read_ios" : 4333182,\n'
                                             '      "write_ios" : 0,\n'
                                             '      "read_merges" : 0,\n'
                                             '      "write_merges" : 0,\n'
                                             '      "read_ticks" : 23213,\n'
                                             '      "write_ticks" : 0,\n'
                                             '      "in_queue" : 23213,\n'
                                             '      "util" : 99.069123\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'libaio-reference: (groupid=0, '
                                             'jobs=1): err= 0: pid=9449: Wed '
                                             'Oct 26 15:20:07 2022\n'
                                             '  read: IOPS=430k, BW=210MiB/s '
                                             '(220MB/s)(1469MiB/7001msec)\n'
                                             '    slat (nsec): min=1442, '
                                             'max=22633, avg=1606.98, '
                                             'stdev=117.24\n'
                                             '    clat (usec): min=2, max=110, '
                                             'avg= 7.47, stdev= 2.67\n'
                                             '     lat (usec): min=6, max=112, '
                                             'avg= 9.08, stdev= 2.67\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 6816],  '
                                             '5.00th=[ 6880], 10.00th=[ 6880], '
                                             '20.00th=[ 6944],\n'
                                             '     | 30.00th=[ 6944], '
                                             '40.00th=[ 6944], 50.00th=[ '
                                             '7008], 60.00th=[ 7008],\n'
                                             '     | 70.00th=[ 7072], '
                                             '80.00th=[ 7200], 90.00th=[ '
                                             '7264], 95.00th=[ 7392],\n'
                                             '     | 99.00th=[22400], '
                                             '99.50th=[27008], '
                                             '99.90th=[32640], '
                                             '99.95th=[35072],\n'
                                             '     | 99.99th=[69120]\n'
                                             '   bw (  KiB/s): min=213880, '
                                             'max=215909, per=100.00%, '
                                             'avg=214911.21, stdev=609.98, '
                                             'samples=14\n'
                                             '   iops        : min=427760, '
                                             'max=431818, avg=429822.43, '
                                             'stdev=1219.97, samples=14\n'
                                             '  lat (usec)   : 4=0.01%, '
                                             '10=96.58%, 20=2.23%, 50=1.15%, '
                                             '100=0.03%\n'
                                             '  lat (usec)   : 250=0.01%\n'
                                             '  cpu          : usr=28.97%, '
                                             'sys=67.20%, ctx=26583, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, 2=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=3008333,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=4\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=210MiB/s (220MB/s), '
                                             '210MiB/s-210MiB/s '
                                             '(220MB/s-220MB/s), io=1469MiB '
                                             '(1540MB), run=7001-7001msec\n'
                                             '\n'
                                             'Disk stats (read/write):\n'
                                             '  nvme0n1: ios=4333182/0, '
                                             'merge=0/0, ticks=23213/0, '
                                             'in_queue=23213, util=99.07%\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_264.output'),
                                   'state': {'begin': 1666783317.6742837,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_4_libaio-reference_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8205270767211914,
                                             'end': 1666783318.4948108,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_264.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_264.state')},
                       'cmd_265': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_512_4_libaio-reference_1.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_265.output'),
                                   'state': {'begin': 1666783319.5633724,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_4_libaio-reference_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8108549118041992,
                                             'end': 1666783320.3742273,
                                             'err': 1,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_265.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_265.state')},
                       'cmd_266': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_266.output'),
                                   'state': {'begin': 1666783320.3770204,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="libaio-reference" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="4" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="libaio" '
                                                    '--filename="/dev/nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_4_libaio-reference_1.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.106910943984985,
                                             'end': 1666783331.4839313,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_266.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_266.state')},
                       'cmd_267': {'output': 'libaio-reference: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=libaio, iodepth=4\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666790421,\n'
                                             '  "timestamp_ms" : '
                                             '1666790421560,\n'
                                             '  "time" : "Wed Oct 26 15:20:21 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"libaio-reference",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"libaio-reference",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "4",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : "libaio",\n'
                                             '        "filename" : '
                                             '"/dev/nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '1547104768,\n'
                                             '        "io_kbytes" : 1510844,\n'
                                             '        "bw_bytes" : 220983397,\n'
                                             '        "bw" : 215804,\n'
                                             '        "iops" : 431607.627482,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 3021685,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1433,\n'
                                             '          "max" : 30357,\n'
                                             '          "mean" : 1591.780563,\n'
                                             '          "stddev" : '
                                             '110.756728,\n'
                                             '          "N" : 3021686\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 4548,\n'
                                             '          "max" : 126658,\n'
                                             '          "mean" : 7446.293310,\n'
                                             '          "stddev" : '
                                             '2622.424351,\n'
                                             '          "N" : 3021688,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 6880,\n'
                                             '            "5.000000" : 6880,\n'
                                             '            "10.000000" : 6944,\n'
                                             '            "20.000000" : 6944,\n'
                                             '            "30.000000" : 6944,\n'
                                             '            "40.000000" : 6944,\n'
                                             '            "50.000000" : 7008,\n'
                                             '            "60.000000" : 7008,\n'
                                             '            "70.000000" : 7008,\n'
                                             '            "80.000000" : 7072,\n'
                                             '            "90.000000" : 7264,\n'
                                             '            "95.000000" : 7392,\n'
                                             '            "99.000000" : '
                                             '21120,\n'
                                             '            "99.500000" : '
                                             '24960,\n'
                                             '            "99.900000" : '
                                             '31872,\n'
                                             '            "99.950000" : '
                                             '35072,\n'
                                             '            "99.990000" : 70144\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 6021,\n'
                                             '          "max" : 128201,\n'
                                             '          "mean" : 9038.073877,\n'
                                             '          "stddev" : '
                                             '2623.736564,\n'
                                             '          "N" : 3021688\n'
                                             '        },\n'
                                             '        "bw_min" : 215056,\n'
                                             '        "bw_max" : 217372,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '215893.642857,\n'
                                             '        "bw_dev" : 579.696420,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 430112,\n'
                                             '        "iops_max" : 434745,\n'
                                             '        "iops_mean" : '
                                             '431787.571429,\n'
                                             '        "iops_stddev" : '
                                             '1159.639978,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 32.028571,\n'
                                             '      "sys_cpu" : 64.071429,\n'
                                             '      "ctx" : 29587,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 96.336084,\n'
                                             '        "20" : 2.499665,\n'
                                             '        "50" : 1.126226,\n'
                                             '        "100" : 0.037760,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 4,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ],\n'
                                             '  "disk_util" : [\n'
                                             '    {\n'
                                             '      "name" : "nvme0n1",\n'
                                             '      "read_ios" : 4350489,\n'
                                             '      "write_ios" : 0,\n'
                                             '      "read_merges" : 0,\n'
                                             '      "write_merges" : 0,\n'
                                             '      "read_ticks" : 21803,\n'
                                             '      "write_ticks" : 0,\n'
                                             '      "in_queue" : 21803,\n'
                                             '      "util" : 99.078934\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'libaio-reference: (groupid=0, '
                                             'jobs=1): err= 0: pid=9497: Wed '
                                             'Oct 26 15:20:21 2022\n'
                                             '  read: IOPS=432k, BW=211MiB/s '
                                             '(221MB/s)(1475MiB/7001msec)\n'
                                             '    slat (nsec): min=1433, '
                                             'max=30357, avg=1591.78, '
                                             'stdev=110.76\n'
                                             '    clat (usec): min=4, max=126, '
                                             'avg= 7.45, stdev= 2.62\n'
                                             '     lat (usec): min=6, max=128, '
                                             'avg= 9.04, stdev= 2.62\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 6880],  '
                                             '5.00th=[ 6880], 10.00th=[ 6944], '
                                             '20.00th=[ 6944],\n'
                                             '     | 30.00th=[ 6944], '
                                             '40.00th=[ 6944], 50.00th=[ '
                                             '7008], 60.00th=[ 7008],\n'
                                             '     | 70.00th=[ 7008], '
                                             '80.00th=[ 7072], 90.00th=[ '
                                             '7264], 95.00th=[ 7392],\n'
                                             '     | 99.00th=[21120], '
                                             '99.50th=[24960], '
                                             '99.90th=[31872], '
                                             '99.95th=[35072],\n'
                                             '     | 99.99th=[70144]\n'
                                             '   bw (  KiB/s): min=215056, '
                                             'max=217372, per=100.00%, '
                                             'avg=215893.64, stdev=579.70, '
                                             'samples=14\n'
                                             '   iops        : min=430112, '
                                             'max=434745, avg=431787.57, '
                                             'stdev=1159.64, samples=14\n'
                                             '  lat (usec)   : 10=96.34%, '
                                             '20=2.50%, 50=1.13%, 100=0.04%, '
                                             '250=0.01%\n'
                                             '  cpu          : usr=32.03%, '
                                             'sys=64.07%, ctx=29587, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, 2=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=3021685,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=4\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=211MiB/s (221MB/s), '
                                             '211MiB/s-211MiB/s '
                                             '(221MB/s-221MB/s), io=1475MiB '
                                             '(1547MB), run=7001-7001msec\n'
                                             '\n'
                                             'Disk stats (read/write):\n'
                                             '  nvme0n1: ios=4350489/0, '
                                             'merge=0/0, ticks=21803/0, '
                                             'in_queue=21803, util=99.08%\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_267.output'),
                                   'state': {'begin': 1666783331.4866912,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_4_libaio-reference_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.869408369064331,
                                             'end': 1666783332.3560996,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_267.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_267.state')},
                       'cmd_268': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_512_4_libaio-reference_2.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_268.output'),
                                   'state': {'begin': 1666783333.4113686,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_4_libaio-reference_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8447458744049072,
                                             'end': 1666783334.2561145,
                                             'err': 1,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_268.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_268.state')},
                       'cmd_269': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_269.output'),
                                   'state': {'begin': 1666783334.2628534,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="libaio-reference" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="4" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="libaio" '
                                                    '--filename="/dev/nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_4_libaio-reference_2.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.113083124160767,
                                             'end': 1666783345.3759365,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_269.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_269.state')},
                       'cmd_27': {'output': 'io_uring_cmd-bdev_xnvme: (g=0): '
                                            'rw=randread, bs=(R) 512B-512B, '
                                            '(W) 512B-512B, (T) 512B-512B, '
                                            'ioengine=spdk_bdev, iodepth=1\n'
                                            'fio-3.32-dirty\n'
                                            'Starting 1 thread\n'
                                            '{\n'
                                            '  "fio version" : '
                                            '"fio-3.32-dirty",\n'
                                            '  "timestamp" : 1666789263,\n'
                                            '  "timestamp_ms" : '
                                            '1666789263383,\n'
                                            '  "time" : "Wed Oct 26 15:01:03 '
                                            '2022",\n'
                                            '  "jobs" : [\n'
                                            '    {\n'
                                            '      "jobname" : '
                                            '"io_uring_cmd-bdev_xnvme",\n'
                                            '      "groupid" : 0,\n'
                                            '      "error" : 0,\n'
                                            '      "eta" : 0,\n'
                                            '      "elapsed" : 11,\n'
                                            '      "job options" : {\n'
                                            '        "name" : '
                                            '"io_uring_cmd-bdev_xnvme",\n'
                                            '        "rw" : "randread",\n'
                                            '        "size" : "12G",\n'
                                            '        "iodepth" : "1",\n'
                                            '        "bs" : "512",\n'
                                            '        "direct" : "0",\n'
                                            '        "thread" : "1",\n'
                                            '        "time_based" : "1",\n'
                                            '        "runtime" : "7",\n'
                                            '        "ramp_time" : "3",\n'
                                            '        "norandommap" : "1",\n'
                                            '        "allow_file_create" : '
                                            '"0",\n'
                                            '        "ioengine" : '
                                            '"spdk_bdev",\n'
                                            '        "spdk_conf" : '
                                            '"/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",\n'
                                            '        "filename" : "Nvme0n1",\n'
                                            '        "cpus_allowed" : "1"\n'
                                            '      },\n'
                                            '      "read" : {\n'
                                            '        "io_bytes" : 507951104,\n'
                                            '        "io_kbytes" : 496046,\n'
                                            '        "bw_bytes" : 72564443,\n'
                                            '        "bw" : 70863,\n'
                                            '        "iops" : 141727.428571,\n'
                                            '        "runtime" : 7000,\n'
                                            '        "total_ios" : 992092,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 250,\n'
                                            '          "max" : 25638,\n'
                                            '          "mean" : 278.643752,\n'
                                            '          "stddev" : 68.503384,\n'
                                            '          "N" : 992092\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 320,\n'
                                            '          "max" : 78928,\n'
                                            '          "mean" : 6484.535887,\n'
                                            '          "stddev" : '
                                            '1090.312041,\n'
                                            '          "N" : 992092,\n'
                                            '          "percentile" : {\n'
                                            '            "1.000000" : 5984,\n'
                                            '            "5.000000" : 6176,\n'
                                            '            "10.000000" : 6240,\n'
                                            '            "20.000000" : 6240,\n'
                                            '            "30.000000" : 6304,\n'
                                            '            "40.000000" : 6304,\n'
                                            '            "50.000000" : 6368,\n'
                                            '            "60.000000" : 6368,\n'
                                            '            "70.000000" : 6432,\n'
                                            '            "80.000000" : 6432,\n'
                                            '            "90.000000" : 6496,\n'
                                            '            "95.000000" : 6624,\n'
                                            '            "99.000000" : 11456,\n'
                                            '            "99.500000" : 13504,\n'
                                            '            "99.900000" : 17024,\n'
                                            '            "99.950000" : 19328,\n'
                                            '            "99.990000" : 58112\n'
                                            '          }\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 5991,\n'
                                            '          "max" : 79199,\n'
                                            '          "mean" : 6763.179639,\n'
                                            '          "stddev" : '
                                            '1092.110784,\n'
                                            '          "N" : 992092\n'
                                            '        },\n'
                                            '        "bw_min" : 70687,\n'
                                            '        "bw_max" : 71173,\n'
                                            '        "bw_agg" : 100.000000,\n'
                                            '        "bw_mean" : '
                                            '70885.071429,\n'
                                            '        "bw_dev" : 141.145023,\n'
                                            '        "bw_samples" : 14,\n'
                                            '        "iops_min" : 141374,\n'
                                            '        "iops_max" : 142346,\n'
                                            '        "iops_mean" : '
                                            '141770.285714,\n'
                                            '        "iops_stddev" : '
                                            '282.276032,\n'
                                            '        "iops_samples" : 14\n'
                                            '      },\n'
                                            '      "write" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "trim" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "sync" : {\n'
                                            '        "total_ios" : 0,\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        }\n'
                                            '      },\n'
                                            '      "job_runtime" : 7000,\n'
                                            '      "usr_cpu" : 99.985714,\n'
                                            '      "sys_cpu" : 0.000000,\n'
                                            '      "ctx" : 10,\n'
                                            '      "majf" : 0,\n'
                                            '      "minf" : 0,\n'
                                            '      "iodepth_level" : {\n'
                                            '        "1" : 100.000000,\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_submit" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_complete" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ns" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.010000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_us" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 98.826016,\n'
                                            '        "20" : 1.129028,\n'
                                            '        "50" : 0.031953,\n'
                                            '        "100" : 0.012398,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ms" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000,\n'
                                            '        "2000" : 0.000000,\n'
                                            '        ">=2000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_depth" : 1,\n'
                                            '      "latency_target" : 0,\n'
                                            '      "latency_percentile" : '
                                            '100.000000,\n'
                                            '      "latency_window" : 0\n'
                                            '    }\n'
                                            '  ]\n'
                                            '}\n'
                                            '\n'
                                            'io_uring_cmd-bdev_xnvme: '
                                            '(groupid=0, jobs=1): err= 0: '
                                            'pid=5343: Wed Oct 26 15:01:03 '
                                            '2022\n'
                                            '  read: IOPS=142k, BW=69.2MiB/s '
                                            '(72.6MB/s)(484MiB/7000msec)\n'
                                            '    slat (nsec): min=250, '
                                            'max=25638, avg=278.64, '
                                            'stdev=68.50\n'
                                            '    clat (nsec): min=320, '
                                            'max=78928, avg=6484.54, '
                                            'stdev=1090.31\n'
                                            '     lat (nsec): min=5991, '
                                            'max=79199, avg=6763.18, '
                                            'stdev=1092.11\n'
                                            '    clat percentiles (nsec):\n'
                                            '     |  1.00th=[ 5984],  5.00th=[ '
                                            '6176], 10.00th=[ 6240], 20.00th=[ '
                                            '6240],\n'
                                            '     | 30.00th=[ 6304], 40.00th=[ '
                                            '6304], 50.00th=[ 6368], 60.00th=[ '
                                            '6368],\n'
                                            '     | 70.00th=[ 6432], 80.00th=[ '
                                            '6432], 90.00th=[ 6496], 95.00th=[ '
                                            '6624],\n'
                                            '     | 99.00th=[11456], '
                                            '99.50th=[13504], 99.90th=[17024], '
                                            '99.95th=[19328],\n'
                                            '     | 99.99th=[58112]\n'
                                            '   bw (  KiB/s): min=70687, '
                                            'max=71173, per=100.00%, '
                                            'avg=70885.07, stdev=141.15, '
                                            'samples=14\n'
                                            '   iops        : min=141374, '
                                            'max=142346, avg=141770.29, '
                                            'stdev=282.28, samples=14\n'
                                            '  lat (nsec)   : 500=0.01%\n'
                                            '  lat (usec)   : 10=98.83%, '
                                            '20=1.13%, 50=0.03%, 100=0.01%\n'
                                            '  cpu          : usr=99.99%, '
                                            'sys=0.00%, ctx=10, majf=0, '
                                            'minf=0\n'
                                            '  IO depths    : 1=100.0%, '
                                            '2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, >=64=0.0%\n'
                                            '     submit    : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     complete  : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     issued rwts: '
                                            'total=992092,0,0,0 short=0,0,0,0 '
                                            'dropped=0,0,0,0\n'
                                            '     latency   : target=0, '
                                            'window=0, percentile=100.00%, '
                                            'depth=1\n'
                                            '\n'
                                            'Run status group 0 (all jobs):\n'
                                            '   READ: bw=69.2MiB/s (72.6MB/s), '
                                            '69.2MiB/s-69.2MiB/s '
                                            '(72.6MB/s-72.6MB/s), io=484MiB '
                                            '(508MB), run=7000-7000msec\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_27.output'),
                                  'state': {'begin': 1666782173.3538754,
                                            'cmd': 'cat '
                                                   '/tmp/fio-output_512_1_io_uring_cmd-bdev_xnvme_2.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.8117146492004395,
                                            'end': 1666782174.16559,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_27.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_27.state')},
                       'cmd_270': {'output': 'libaio-reference: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=libaio, iodepth=4\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666790435,\n'
                                             '  "timestamp_ms" : '
                                             '1666790435451,\n'
                                             '  "time" : "Wed Oct 26 15:20:35 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"libaio-reference",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"libaio-reference",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "4",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : "libaio",\n'
                                             '        "filename" : '
                                             '"/dev/nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '1523316736,\n'
                                             '        "io_kbytes" : 1487614,\n'
                                             '        "bw_bytes" : 217585592,\n'
                                             '        "bw" : 212485,\n'
                                             '        "iops" : 424971.289816,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 2975224,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1453,\n'
                                             '          "max" : 26480,\n'
                                             '          "mean" : 1626.616954,\n'
                                             '          "stddev" : '
                                             '111.977939,\n'
                                             '          "N" : 2975225\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 4588,\n'
                                             '          "max" : 122941,\n'
                                             '          "mean" : 7550.090039,\n'
                                             '          "stddev" : '
                                             '2701.313923,\n'
                                             '          "N" : 2975227,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 6944,\n'
                                             '            "5.000000" : 6944,\n'
                                             '            "10.000000" : 7008,\n'
                                             '            "20.000000" : 7008,\n'
                                             '            "30.000000" : 7008,\n'
                                             '            "40.000000" : 7072,\n'
                                             '            "50.000000" : 7072,\n'
                                             '            "60.000000" : 7072,\n'
                                             '            "70.000000" : 7136,\n'
                                             '            "80.000000" : 7328,\n'
                                             '            "90.000000" : 7392,\n'
                                             '            "95.000000" : 7456,\n'
                                             '            "99.000000" : '
                                             '22400,\n'
                                             '            "99.500000" : '
                                             '27008,\n'
                                             '            "99.900000" : '
                                             '33024,\n'
                                             '            "99.950000" : '
                                             '38656,\n'
                                             '            "99.990000" : 71168\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 6101,\n'
                                             '          "max" : 124514,\n'
                                             '          "mean" : 9176.707014,\n'
                                             '          "stddev" : '
                                             '2705.260929,\n'
                                             '          "N" : 2975227\n'
                                             '        },\n'
                                             '        "bw_min" : 212046,\n'
                                             '        "bw_max" : 213823,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '212606.928571,\n'
                                             '        "bw_dev" : 548.418771,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 424092,\n'
                                             '        "iops_max" : 427646,\n'
                                             '        "iops_mean" : '
                                             '425213.928571,\n'
                                             '        "iops_stddev" : '
                                             '1096.881206,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 34.171429,\n'
                                             '      "sys_cpu" : 62.028571,\n'
                                             '      "ctx" : 26352,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 96.603953,\n'
                                             '        "20" : 2.192776,\n'
                                             '        "50" : 1.162669,\n'
                                             '        "100" : 0.040568,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 4,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ],\n'
                                             '  "disk_util" : [\n'
                                             '    {\n'
                                             '      "name" : "nvme0n1",\n'
                                             '      "read_ios" : 4292890,\n'
                                             '      "write_ios" : 0,\n'
                                             '      "read_merges" : 0,\n'
                                             '      "write_merges" : 0,\n'
                                             '      "read_ticks" : 22927,\n'
                                             '      "write_ticks" : 0,\n'
                                             '      "in_queue" : 22927,\n'
                                             '      "util" : 99.088748\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'libaio-reference: (groupid=0, '
                                             'jobs=1): err= 0: pid=9543: Wed '
                                             'Oct 26 15:20:35 2022\n'
                                             '  read: IOPS=425k, BW=208MiB/s '
                                             '(218MB/s)(1453MiB/7001msec)\n'
                                             '    slat (nsec): min=1453, '
                                             'max=26480, avg=1626.62, '
                                             'stdev=111.98\n'
                                             '    clat (usec): min=4, max=122, '
                                             'avg= 7.55, stdev= 2.70\n'
                                             '     lat (usec): min=6, max=124, '
                                             'avg= 9.18, stdev= 2.71\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 6944],  '
                                             '5.00th=[ 6944], 10.00th=[ 7008], '
                                             '20.00th=[ 7008],\n'
                                             '     | 30.00th=[ 7008], '
                                             '40.00th=[ 7072], 50.00th=[ '
                                             '7072], 60.00th=[ 7072],\n'
                                             '     | 70.00th=[ 7136], '
                                             '80.00th=[ 7328], 90.00th=[ '
                                             '7392], 95.00th=[ 7456],\n'
                                             '     | 99.00th=[22400], '
                                             '99.50th=[27008], '
                                             '99.90th=[33024], '
                                             '99.95th=[38656],\n'
                                             '     | 99.99th=[71168]\n'
                                             '   bw (  KiB/s): min=212046, '
                                             'max=213823, per=100.00%, '
                                             'avg=212606.93, stdev=548.42, '
                                             'samples=14\n'
                                             '   iops        : min=424092, '
                                             'max=427646, avg=425213.93, '
                                             'stdev=1096.88, samples=14\n'
                                             '  lat (usec)   : 10=96.60%, '
                                             '20=2.19%, 50=1.16%, 100=0.04%, '
                                             '250=0.01%\n'
                                             '  cpu          : usr=34.17%, '
                                             'sys=62.03%, ctx=26352, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, 2=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=2975224,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=4\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=208MiB/s (218MB/s), '
                                             '208MiB/s-208MiB/s '
                                             '(218MB/s-218MB/s), io=1453MiB '
                                             '(1523MB), run=7001-7001msec\n'
                                             '\n'
                                             'Disk stats (read/write):\n'
                                             '  nvme0n1: ios=4292890/0, '
                                             'merge=0/0, ticks=22927/0, '
                                             'in_queue=22927, util=99.09%\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_270.output'),
                                   'state': {'begin': 1666783345.3784652,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_4_libaio-reference_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8766143321990967,
                                             'end': 1666783346.2550795,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_270.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_270.state')},
                       'cmd_271': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_512_4_libaio-bdev_aio_0.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_271.output'),
                                   'state': {'begin': 1666783348.498997,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_4_libaio-bdev_aio_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.806602954864502,
                                             'end': 1666783349.3056,
                                             'err': 1,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_271.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_271.state')},
                       'cmd_272': {'output': 'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_272.output'),
                                   'state': {'begin': 1666783349.307994,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="libaio-bdev_aio" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="4" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="spdk_bdev" '
                                                    '--spdk_conf="/tmp/spdk_bdev_aio_foo.conf" '
                                                    '--filename="Nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_4_libaio-bdev_aio_0.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.4180166721344,
                                             'end': 1666783360.7260106,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_272.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_272.state')},
                       'cmd_273': {'output': 'libaio-bdev_aio: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=spdk_bdev, iodepth=4\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666790450,\n'
                                             '  "timestamp_ms" : '
                                             '1666790450686,\n'
                                             '  "time" : "Wed Oct 26 15:20:50 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"libaio-bdev_aio",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"libaio-bdev_aio",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "4",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"spdk_bdev",\n'
                                             '        "spdk_conf" : '
                                             '"/tmp/spdk_bdev_aio_foo.conf",\n'
                                             '        "filename" : "Nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '1562015232,\n'
                                             '        "io_kbytes" : 1525405,\n'
                                             '        "bw_bytes" : 223113159,\n'
                                             '        "bw" : 217883,\n'
                                             '        "iops" : 435767.318954,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 3050807,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1623,\n'
                                             '          "max" : 24445,\n'
                                             '          "mean" : 1823.933966,\n'
                                             '          "stddev" : '
                                             '122.188443,\n'
                                             '          "N" : 3050808\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 271,\n'
                                             '          "max" : 121558,\n'
                                             '          "mean" : 6479.020192,\n'
                                             '          "stddev" : '
                                             '2349.697327,\n'
                                             '          "N" : 3050807,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 4384,\n'
                                             '            "5.000000" : 4448,\n'
                                             '            "10.000000" : 4576,\n'
                                             '            "20.000000" : 4704,\n'
                                             '            "30.000000" : 6496,\n'
                                             '            "40.000000" : 6624,\n'
                                             '            "50.000000" : 6688,\n'
                                             '            "60.000000" : 6752,\n'
                                             '            "70.000000" : 6752,\n'
                                             '            "80.000000" : 6816,\n'
                                             '            "90.000000" : 6944,\n'
                                             '            "95.000000" : 7136,\n'
                                             '            "99.000000" : '
                                             '17536,\n'
                                             '            "99.500000" : '
                                             '20352,\n'
                                             '            "99.900000" : '
                                             '25472,\n'
                                             '            "99.950000" : '
                                             '30848,\n'
                                             '            "99.990000" : 65280\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 5370,\n'
                                             '          "max" : 123371,\n'
                                             '          "mean" : 8302.954171,\n'
                                             '          "stddev" : '
                                             '2352.126482,\n'
                                             '          "N" : 3050807\n'
                                             '        },\n'
                                             '        "bw_min" : 216213,\n'
                                             '        "bw_max" : 219913,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '218014.214286,\n'
                                             '        "bw_dev" : 1097.446559,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 432426,\n'
                                             '        "iops_max" : 439826,\n'
                                             '        "iops_mean" : '
                                             '436028.714286,\n'
                                             '        "iops_stddev" : '
                                             '2194.874043,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 38.500000,\n'
                                             '      "sys_cpu" : 61.471429,\n'
                                             '      "ctx" : 14,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.100000,\n'
                                             '        "2" : 29.427558,\n'
                                             '        "4" : 70.569525,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.010000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 96.818940,\n'
                                             '        "20" : 2.569976,\n'
                                             '        "50" : 0.567129,\n'
                                             '        "100" : 0.034483,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 4,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'libaio-bdev_aio: (groupid=0, '
                                             'jobs=1): err= 0: pid=9600: Wed '
                                             'Oct 26 15:20:50 2022\n'
                                             '  read: IOPS=436k, BW=213MiB/s '
                                             '(223MB/s)(1490MiB/7001msec)\n'
                                             '    slat (nsec): min=1623, '
                                             'max=24445, avg=1823.93, '
                                             'stdev=122.19\n'
                                             '    clat (nsec): min=271, '
                                             'max=121558, avg=6479.02, '
                                             'stdev=2349.70\n'
                                             '     lat (usec): min=5, max=123, '
                                             'avg= 8.30, stdev= 2.35\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 4384],  '
                                             '5.00th=[ 4448], 10.00th=[ 4576], '
                                             '20.00th=[ 4704],\n'
                                             '     | 30.00th=[ 6496], '
                                             '40.00th=[ 6624], 50.00th=[ '
                                             '6688], 60.00th=[ 6752],\n'
                                             '     | 70.00th=[ 6752], '
                                             '80.00th=[ 6816], 90.00th=[ '
                                             '6944], 95.00th=[ 7136],\n'
                                             '     | 99.00th=[17536], '
                                             '99.50th=[20352], '
                                             '99.90th=[25472], '
                                             '99.95th=[30848],\n'
                                             '     | 99.99th=[65280]\n'
                                             '   bw (  KiB/s): min=216213, '
                                             'max=219913, per=100.00%, '
                                             'avg=218014.21, stdev=1097.45, '
                                             'samples=14\n'
                                             '   iops        : min=432426, '
                                             'max=439826, avg=436028.71, '
                                             'stdev=2194.87, samples=14\n'
                                             '  lat (nsec)   : 500=0.01%\n'
                                             '  lat (usec)   : 4=0.01%, '
                                             '10=96.82%, 20=2.57%, 50=0.57%, '
                                             '100=0.03%\n'
                                             '  lat (usec)   : 250=0.01%\n'
                                             '  cpu          : usr=38.50%, '
                                             'sys=61.47%, ctx=14, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.1%, '
                                             '2=29.4%, 4=70.6%, 8=0.0%, '
                                             '16=0.0%, 32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=3050807,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=4\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=213MiB/s (223MB/s), '
                                             '213MiB/s-213MiB/s '
                                             '(223MB/s-223MB/s), io=1490MiB '
                                             '(1562MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_273.output'),
                                   'state': {'begin': 1666783360.7284133,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_4_libaio-bdev_aio_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8341774940490723,
                                             'end': 1666783361.5625908,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_273.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_273.state')},
                       'cmd_274': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_512_4_libaio-bdev_aio_1.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_274.output'),
                                   'state': {'begin': 1666783363.6808743,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_4_libaio-bdev_aio_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8134632110595703,
                                             'end': 1666783364.4943376,
                                             'err': 1,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_274.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_274.state')},
                       'cmd_275': {'output': 'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_275.output'),
                                   'state': {'begin': 1666783364.4964976,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="libaio-bdev_aio" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="4" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="spdk_bdev" '
                                                    '--spdk_conf="/tmp/spdk_bdev_aio_foo.conf" '
                                                    '--filename="Nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_4_libaio-bdev_aio_1.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.582042932510376,
                                             'end': 1666783376.0785406,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_275.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_275.state')},
                       'cmd_276': {'output': 'libaio-bdev_aio: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=spdk_bdev, iodepth=4\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666790466,\n'
                                             '  "timestamp_ms" : '
                                             '1666790466054,\n'
                                             '  "time" : "Wed Oct 26 15:21:06 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"libaio-bdev_aio",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"libaio-bdev_aio",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "4",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"spdk_bdev",\n'
                                             '        "spdk_conf" : '
                                             '"/tmp/spdk_bdev_aio_foo.conf",\n'
                                             '        "filename" : "Nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '1556518912,\n'
                                             '        "io_kbytes" : 1520038,\n'
                                             '        "bw_bytes" : 222328083,\n'
                                             '        "bw" : 217117,\n'
                                             '        "iops" : 434234.109413,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 3040073,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1643,\n'
                                             '          "max" : 40897,\n'
                                             '          "mean" : 1833.893626,\n'
                                             '          "stddev" : '
                                             '124.135208,\n'
                                             '          "N" : 3040073\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 290,\n'
                                             '          "max" : 124533,\n'
                                             '          "mean" : 6469.387454,\n'
                                             '          "stddev" : '
                                             '2382.210011,\n'
                                             '          "N" : 3040076,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 4448,\n'
                                             '            "5.000000" : 4448,\n'
                                             '            "10.000000" : 4576,\n'
                                             '            "20.000000" : 4704,\n'
                                             '            "30.000000" : 6432,\n'
                                             '            "40.000000" : 6624,\n'
                                             '            "50.000000" : 6688,\n'
                                             '            "60.000000" : 6752,\n'
                                             '            "70.000000" : 6816,\n'
                                             '            "80.000000" : 6816,\n'
                                             '            "90.000000" : 6944,\n'
                                             '            "95.000000" : 7200,\n'
                                             '            "99.000000" : '
                                             '17536,\n'
                                             '            "99.500000" : '
                                             '20352,\n'
                                             '            "99.900000" : '
                                             '25728,\n'
                                             '            "99.950000" : '
                                             '31872,\n'
                                             '            "99.990000" : 66048\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 5440,\n'
                                             '          "max" : 126347,\n'
                                             '          "mean" : 8303.281079,\n'
                                             '          "stddev" : '
                                             '2384.467555,\n'
                                             '          "N" : 3040076\n'
                                             '        },\n'
                                             '        "bw_min" : 216236,\n'
                                             '        "bw_max" : 218096,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '217165.357143,\n'
                                             '        "bw_dev" : 551.447969,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 432472,\n'
                                             '        "iops_max" : 436190,\n'
                                             '        "iops_mean" : '
                                             '434330.785714,\n'
                                             '        "iops_stddev" : '
                                             '1102.928562,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 39.585714,\n'
                                             '      "sys_cpu" : 60.385714,\n'
                                             '      "ctx" : 14,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.100000,\n'
                                             '        "2" : 31.062050,\n'
                                             '        "4" : 68.935220,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.010000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.010000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 96.805800,\n'
                                             '        "20" : 2.581681,\n'
                                             '        "50" : 0.566894,\n'
                                             '        "100" : 0.036775,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 4,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'libaio-bdev_aio: (groupid=0, '
                                             'jobs=1): err= 0: pid=9659: Wed '
                                             'Oct 26 15:21:06 2022\n'
                                             '  read: IOPS=434k, BW=212MiB/s '
                                             '(222MB/s)(1484MiB/7001msec)\n'
                                             '    slat (nsec): min=1643, '
                                             'max=40897, avg=1833.89, '
                                             'stdev=124.14\n'
                                             '    clat (nsec): min=290, '
                                             'max=124533, avg=6469.39, '
                                             'stdev=2382.21\n'
                                             '     lat (usec): min=5, max=126, '
                                             'avg= 8.30, stdev= 2.38\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 4448],  '
                                             '5.00th=[ 4448], 10.00th=[ 4576], '
                                             '20.00th=[ 4704],\n'
                                             '     | 30.00th=[ 6432], '
                                             '40.00th=[ 6624], 50.00th=[ '
                                             '6688], 60.00th=[ 6752],\n'
                                             '     | 70.00th=[ 6816], '
                                             '80.00th=[ 6816], 90.00th=[ '
                                             '6944], 95.00th=[ 7200],\n'
                                             '     | 99.00th=[17536], '
                                             '99.50th=[20352], '
                                             '99.90th=[25728], '
                                             '99.95th=[31872],\n'
                                             '     | 99.99th=[66048]\n'
                                             '   bw (  KiB/s): min=216236, '
                                             'max=218096, per=100.00%, '
                                             'avg=217165.36, stdev=551.45, '
                                             'samples=14\n'
                                             '   iops        : min=432472, '
                                             'max=436190, avg=434330.79, '
                                             'stdev=1102.93, samples=14\n'
                                             '  lat (nsec)   : 500=0.01%\n'
                                             '  lat (usec)   : 2=0.01%, '
                                             '4=0.01%, 10=96.81%, 20=2.58%, '
                                             '50=0.57%\n'
                                             '  lat (usec)   : 100=0.04%, '
                                             '250=0.01%\n'
                                             '  cpu          : usr=39.59%, '
                                             'sys=60.39%, ctx=14, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.1%, '
                                             '2=31.1%, 4=68.9%, 8=0.0%, '
                                             '16=0.0%, 32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=3040073,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=4\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=212MiB/s (222MB/s), '
                                             '212MiB/s-212MiB/s '
                                             '(222MB/s-222MB/s), io=1484MiB '
                                             '(1557MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_276.output'),
                                   'state': {'begin': 1666783376.0811865,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_4_libaio-bdev_aio_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.830763578414917,
                                             'end': 1666783376.91195,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_276.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_276.state')},
                       'cmd_277': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_512_4_libaio-bdev_aio_2.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_277.output'),
                                   'state': {'begin': 1666783379.0835202,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_4_libaio-bdev_aio_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8354787826538086,
                                             'end': 1666783379.918999,
                                             'err': 1,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_277.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_277.state')},
                       'cmd_278': {'output': 'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_278.output'),
                                   'state': {'begin': 1666783379.9209187,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="libaio-bdev_aio" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="4" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="spdk_bdev" '
                                                    '--spdk_conf="/tmp/spdk_bdev_aio_foo.conf" '
                                                    '--filename="Nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_4_libaio-bdev_aio_2.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.53004240989685,
                                             'end': 1666783391.450961,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_278.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_278.state')},
                       'cmd_279': {'output': 'libaio-bdev_aio: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=spdk_bdev, iodepth=4\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666790481,\n'
                                             '  "timestamp_ms" : '
                                             '1666790481430,\n'
                                             '  "time" : "Wed Oct 26 15:21:21 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"libaio-bdev_aio",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"libaio-bdev_aio",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "4",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"spdk_bdev",\n'
                                             '        "spdk_conf" : '
                                             '"/tmp/spdk_bdev_aio_foo.conf",\n'
                                             '        "filename" : "Nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '1547431936,\n'
                                             '        "io_kbytes" : 1511164,\n'
                                             '        "bw_bytes" : 221030129,\n'
                                             '        "bw" : 215849,\n'
                                             '        "iops" : 431699.042994,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 3022325,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1643,\n'
                                             '          "max" : 40075,\n'
                                             '          "mean" : 1835.594190,\n'
                                             '          "stddev" : '
                                             '126.051482,\n'
                                             '          "N" : 3022325\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 300,\n'
                                             '          "max" : 120215,\n'
                                             '          "mean" : 6500.137739,\n'
                                             '          "stddev" : '
                                             '2390.829741,\n'
                                             '          "N" : 3022328,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 4448,\n'
                                             '            "5.000000" : 4512,\n'
                                             '            "10.000000" : 4576,\n'
                                             '            "20.000000" : 4704,\n'
                                             '            "30.000000" : 6432,\n'
                                             '            "40.000000" : 6688,\n'
                                             '            "50.000000" : 6752,\n'
                                             '            "60.000000" : 6752,\n'
                                             '            "70.000000" : 6816,\n'
                                             '            "80.000000" : 6880,\n'
                                             '            "90.000000" : 7072,\n'
                                             '            "95.000000" : 7264,\n'
                                             '            "99.000000" : '
                                             '17536,\n'
                                             '            "99.500000" : '
                                             '20608,\n'
                                             '            "99.900000" : '
                                             '25984,\n'
                                             '            "99.950000" : '
                                             '32128,\n'
                                             '            "99.990000" : 64768\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 5340,\n'
                                             '          "max" : 122139,\n'
                                             '          "mean" : 8335.731884,\n'
                                             '          "stddev" : '
                                             '2394.300500,\n'
                                             '          "N" : 3022328\n'
                                             '        },\n'
                                             '        "bw_min" : 215090,\n'
                                             '        "bw_max" : 216619,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '215940.769231,\n'
                                             '        "bw_dev" : 480.123969,\n'
                                             '        "bw_samples" : 13,\n'
                                             '        "iops_min" : 430180,\n'
                                             '        "iops_max" : 433238,\n'
                                             '        "iops_mean" : '
                                             '431881.461538,\n'
                                             '        "iops_stddev" : '
                                             '960.297056,\n'
                                             '        "iops_samples" : 13\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 40.442857,\n'
                                             '      "sys_cpu" : 59.528571,\n'
                                             '      "ctx" : 15,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.100000,\n'
                                             '        "2" : 31.311093,\n'
                                             '        "4" : 68.686789,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.010000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 96.840710,\n'
                                             '        "20" : 2.537020,\n'
                                             '        "50" : 0.573764,\n'
                                             '        "100" : 0.037653,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 4,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'libaio-bdev_aio: (groupid=0, '
                                             'jobs=1): err= 0: pid=9716: Wed '
                                             'Oct 26 15:21:21 2022\n'
                                             '  read: IOPS=432k, BW=211MiB/s '
                                             '(221MB/s)(1476MiB/7001msec)\n'
                                             '    slat (nsec): min=1643, '
                                             'max=40075, avg=1835.59, '
                                             'stdev=126.05\n'
                                             '    clat (nsec): min=300, '
                                             'max=120215, avg=6500.14, '
                                             'stdev=2390.83\n'
                                             '     lat (usec): min=5, max=122, '
                                             'avg= 8.34, stdev= 2.39\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 4448],  '
                                             '5.00th=[ 4512], 10.00th=[ 4576], '
                                             '20.00th=[ 4704],\n'
                                             '     | 30.00th=[ 6432], '
                                             '40.00th=[ 6688], 50.00th=[ '
                                             '6752], 60.00th=[ 6752],\n'
                                             '     | 70.00th=[ 6816], '
                                             '80.00th=[ 6880], 90.00th=[ '
                                             '7072], 95.00th=[ 7264],\n'
                                             '     | 99.00th=[17536], '
                                             '99.50th=[20608], '
                                             '99.90th=[25984], '
                                             '99.95th=[32128],\n'
                                             '     | 99.99th=[64768]\n'
                                             '   bw (  KiB/s): min=215090, '
                                             'max=216619, per=100.00%, '
                                             'avg=215940.77, stdev=480.12, '
                                             'samples=13\n'
                                             '   iops        : min=430180, '
                                             'max=433238, avg=431881.46, '
                                             'stdev=960.30, samples=13\n'
                                             '  lat (nsec)   : 500=0.01%\n'
                                             '  lat (usec)   : 4=0.01%, '
                                             '10=96.84%, 20=2.54%, 50=0.57%, '
                                             '100=0.04%\n'
                                             '  lat (usec)   : 250=0.01%\n'
                                             '  cpu          : usr=40.44%, '
                                             'sys=59.53%, ctx=15, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.1%, '
                                             '2=31.3%, 4=68.7%, 8=0.0%, '
                                             '16=0.0%, 32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=3022325,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=4\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=211MiB/s (221MB/s), '
                                             '211MiB/s-211MiB/s '
                                             '(221MB/s-221MB/s), io=1476MiB '
                                             '(1547MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_279.output'),
                                   'state': {'begin': 1666783391.4535267,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_4_libaio-bdev_aio_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8779900074005127,
                                             'end': 1666783392.3315167,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_279.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_279.state')},
                       'cmd_28': {'output': 'rm: cannot remove '
                                            "'/tmp/fio-output_512_1_io_uring-reference_0.txt': "
                                            'No such file or directory\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_28.output'),
                                  'state': {'begin': 1666782175.175131,
                                            'cmd': 'rm '
                                                   '/tmp/fio-output_512_1_io_uring-reference_0.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.8835101127624512,
                                            'end': 1666782176.0586412,
                                            'err': 1,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_28.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_28.state')},
                       'cmd_280': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_512_4_libaio-bdev_xnvme_0.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_280.output'),
                                   'state': {'begin': 1666783394.416069,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_4_libaio-bdev_xnvme_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7993080615997314,
                                             'end': 1666783395.215377,
                                             'err': 1,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_280.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_280.state')},
                       'cmd_281': {'output': 'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_281.output'),
                                   'state': {'begin': 1666783395.2174687,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="libaio-bdev_xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="4" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="spdk_bdev" '
                                                    '--spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" '
                                                    '--filename="Nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_4_libaio-bdev_xnvme_0.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.530804872512817,
                                             'end': 1666783406.7482736,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_281.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_281.state')},
                       'cmd_282': {'output': 'libaio-bdev_xnvme: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=spdk_bdev, iodepth=4\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666790496,\n'
                                             '  "timestamp_ms" : '
                                             '1666790496711,\n'
                                             '  "time" : "Wed Oct 26 15:21:36 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"libaio-bdev_xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"libaio-bdev_xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "4",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"spdk_bdev",\n'
                                             '        "spdk_conf" : '
                                             '"/tmp/spdk_bdev_xnvme_libaio.conf",\n'
                                             '        "filename" : "Nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '1444080640,\n'
                                             '        "io_kbytes" : 1410235,\n'
                                             '        "bw_bytes" : 206267767,\n'
                                             '        "bw" : 201433,\n'
                                             '        "iops" : 402866.161977,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 2820466,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1673,\n'
                                             '          "max" : 34555,\n'
                                             '          "mean" : 1859.775083,\n'
                                             '          "stddev" : '
                                             '123.879869,\n'
                                             '          "N" : 2820467\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 701,\n'
                                             '          "max" : 115096,\n'
                                             '          "mean" : 6746.611971,\n'
                                             '          "stddev" : '
                                             '2343.713145,\n'
                                             '          "N" : 2820466,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 5216,\n'
                                             '            "5.000000" : 5280,\n'
                                             '            "10.000000" : 5280,\n'
                                             '            "20.000000" : 5344,\n'
                                             '            "30.000000" : 5408,\n'
                                             '            "40.000000" : 5472,\n'
                                             '            "50.000000" : 7264,\n'
                                             '            "60.000000" : 7328,\n'
                                             '            "70.000000" : 7392,\n'
                                             '            "80.000000" : 7456,\n'
                                             '            "90.000000" : 7584,\n'
                                             '            "95.000000" : 7968,\n'
                                             '            "99.000000" : '
                                             '17536,\n'
                                             '            "99.500000" : '
                                             '20608,\n'
                                             '            "99.900000" : '
                                             '26240,\n'
                                             '            "99.950000" : '
                                             '32128,\n'
                                             '            "99.990000" : 63744\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 5801,\n'
                                             '          "max" : 116919,\n'
                                             '          "mean" : 8606.387073,\n'
                                             '          "stddev" : '
                                             '2346.267108,\n'
                                             '          "N" : 2820466\n'
                                             '        },\n'
                                             '        "bw_min" : 200600,\n'
                                             '        "bw_max" : 202229,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '201469.857143,\n'
                                             '        "bw_dev" : 500.196093,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 401200,\n'
                                             '        "iops_max" : 404458,\n'
                                             '        "iops_mean" : '
                                             '402939.714286,\n'
                                             '        "iops_stddev" : '
                                             '1000.392187,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 37.757143,\n'
                                             '      "sys_cpu" : 62.214286,\n'
                                             '      "ctx" : 14,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.100000,\n'
                                             '        "2" : 49.370813,\n'
                                             '        "4" : 50.626563,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.010000,\n'
                                             '        "1000" : 0.010000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 96.743020,\n'
                                             '        "20" : 2.692959,\n'
                                             '        "50" : 0.519382,\n'
                                             '        "100" : 0.038150,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 4,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'libaio-bdev_xnvme: (groupid=0, '
                                             'jobs=1): err= 0: pid=9774: Wed '
                                             'Oct 26 15:21:36 2022\n'
                                             '  read: IOPS=403k, BW=197MiB/s '
                                             '(206MB/s)(1377MiB/7001msec)\n'
                                             '    slat (nsec): min=1673, '
                                             'max=34555, avg=1859.78, '
                                             'stdev=123.88\n'
                                             '    clat (nsec): min=701, '
                                             'max=115096, avg=6746.61, '
                                             'stdev=2343.71\n'
                                             '     lat (usec): min=5, max=116, '
                                             'avg= 8.61, stdev= 2.35\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 5216],  '
                                             '5.00th=[ 5280], 10.00th=[ 5280], '
                                             '20.00th=[ 5344],\n'
                                             '     | 30.00th=[ 5408], '
                                             '40.00th=[ 5472], 50.00th=[ '
                                             '7264], 60.00th=[ 7328],\n'
                                             '     | 70.00th=[ 7392], '
                                             '80.00th=[ 7456], 90.00th=[ '
                                             '7584], 95.00th=[ 7968],\n'
                                             '     | 99.00th=[17536], '
                                             '99.50th=[20608], '
                                             '99.90th=[26240], '
                                             '99.95th=[32128],\n'
                                             '     | 99.99th=[63744]\n'
                                             '   bw (  KiB/s): min=200600, '
                                             'max=202229, per=100.00%, '
                                             'avg=201469.86, stdev=500.20, '
                                             'samples=14\n'
                                             '   iops        : min=401200, '
                                             'max=404458, avg=402939.71, '
                                             'stdev=1000.39, samples=14\n'
                                             '  lat (nsec)   : 750=0.01%, '
                                             '1000=0.01%\n'
                                             '  lat (usec)   : 4=0.01%, '
                                             '10=96.74%, 20=2.69%, 50=0.52%, '
                                             '100=0.04%\n'
                                             '  lat (usec)   : 250=0.01%\n'
                                             '  cpu          : usr=37.76%, '
                                             'sys=62.21%, ctx=14, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.1%, '
                                             '2=49.4%, 4=50.6%, 8=0.0%, '
                                             '16=0.0%, 32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=2820466,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=4\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=197MiB/s (206MB/s), '
                                             '197MiB/s-197MiB/s '
                                             '(206MB/s-206MB/s), io=1377MiB '
                                             '(1444MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_282.output'),
                                   'state': {'begin': 1666783406.7510893,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_4_libaio-bdev_xnvme_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8057091236114502,
                                             'end': 1666783407.5567985,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_282.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_282.state')},
                       'cmd_283': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_512_4_libaio-bdev_xnvme_1.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_283.output'),
                                   'state': {'begin': 1666783409.6616611,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_4_libaio-bdev_xnvme_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8075447082519531,
                                             'end': 1666783410.4692059,
                                             'err': 1,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_283.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_283.state')},
                       'cmd_284': {'output': 'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_284.output'),
                                   'state': {'begin': 1666783410.4711049,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="libaio-bdev_xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="4" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="spdk_bdev" '
                                                    '--spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" '
                                                    '--filename="Nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_4_libaio-bdev_xnvme_1.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.679587841033936,
                                             'end': 1666783422.1506927,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_284.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_284.state')},
                       'cmd_285': {'output': 'libaio-bdev_xnvme: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=spdk_bdev, iodepth=4\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666790512,\n'
                                             '  "timestamp_ms" : '
                                             '1666790512099,\n'
                                             '  "time" : "Wed Oct 26 15:21:52 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"libaio-bdev_xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"libaio-bdev_xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "4",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"spdk_bdev",\n'
                                             '        "spdk_conf" : '
                                             '"/tmp/spdk_bdev_xnvme_libaio.conf",\n'
                                             '        "filename" : "Nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '1448945664,\n'
                                             '        "io_kbytes" : 1414986,\n'
                                             '        "bw_bytes" : 206962671,\n'
                                             '        "bw" : 202111,\n'
                                             '        "iops" : 404223.539494,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 2829969,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1653,\n'
                                             '          "max" : 22372,\n'
                                             '          "mean" : 1839.170135,\n'
                                             '          "stddev" : '
                                             '121.423768,\n'
                                             '          "N" : 2829970\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 691,\n'
                                             '          "max" : 120626,\n'
                                             '          "mean" : 6717.754872,\n'
                                             '          "stddev" : '
                                             '2347.788874,\n'
                                             '          "N" : 2829972,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 5152,\n'
                                             '            "5.000000" : 5280,\n'
                                             '            "10.000000" : 5280,\n'
                                             '            "20.000000" : 5280,\n'
                                             '            "30.000000" : 5344,\n'
                                             '            "40.000000" : 5408,\n'
                                             '            "50.000000" : 7264,\n'
                                             '            "60.000000" : 7328,\n'
                                             '            "70.000000" : 7328,\n'
                                             '            "80.000000" : 7392,\n'
                                             '            "90.000000" : 7584,\n'
                                             '            "95.000000" : 7968,\n'
                                             '            "99.000000" : '
                                             '17536,\n'
                                             '            "99.500000" : '
                                             '20608,\n'
                                             '            "99.900000" : '
                                             '26240,\n'
                                             '            "99.950000" : '
                                             '32128,\n'
                                             '            "99.990000" : 64256\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 5520,\n'
                                             '          "max" : 122500,\n'
                                             '          "mean" : 8556.924986,\n'
                                             '          "stddev" : '
                                             '2350.299146,\n'
                                             '          "N" : 2829972\n'
                                             '        },\n'
                                             '        "bw_min" : 201099,\n'
                                             '        "bw_max" : 203576,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '202148.357143,\n'
                                             '        "bw_dev" : 804.850259,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 402198,\n'
                                             '        "iops_max" : 407152,\n'
                                             '        "iops_mean" : '
                                             '404296.714286,\n'
                                             '        "iops_stddev" : '
                                             '1609.700518,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 39.814286,\n'
                                             '      "sys_cpu" : 60.157143,\n'
                                             '      "ctx" : 12,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.100000,\n'
                                             '        "2" : 49.212165,\n'
                                             '        "4" : 50.785291,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.010000,\n'
                                             '        "1000" : 0.010000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 96.734523,\n'
                                             '        "20" : 2.704376,\n'
                                             '        "50" : 0.516825,\n'
                                             '        "100" : 0.037209,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 4,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'libaio-bdev_xnvme: (groupid=0, '
                                             'jobs=1): err= 0: pid=9831: Wed '
                                             'Oct 26 15:21:52 2022\n'
                                             '  read: IOPS=404k, BW=197MiB/s '
                                             '(207MB/s)(1382MiB/7001msec)\n'
                                             '    slat (nsec): min=1653, '
                                             'max=22372, avg=1839.17, '
                                             'stdev=121.42\n'
                                             '    clat (nsec): min=691, '
                                             'max=120626, avg=6717.75, '
                                             'stdev=2347.79\n'
                                             '     lat (usec): min=5, max=122, '
                                             'avg= 8.56, stdev= 2.35\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 5152],  '
                                             '5.00th=[ 5280], 10.00th=[ 5280], '
                                             '20.00th=[ 5280],\n'
                                             '     | 30.00th=[ 5344], '
                                             '40.00th=[ 5408], 50.00th=[ '
                                             '7264], 60.00th=[ 7328],\n'
                                             '     | 70.00th=[ 7328], '
                                             '80.00th=[ 7392], 90.00th=[ '
                                             '7584], 95.00th=[ 7968],\n'
                                             '     | 99.00th=[17536], '
                                             '99.50th=[20608], '
                                             '99.90th=[26240], '
                                             '99.95th=[32128],\n'
                                             '     | 99.99th=[64256]\n'
                                             '   bw (  KiB/s): min=201099, '
                                             'max=203576, per=100.00%, '
                                             'avg=202148.36, stdev=804.85, '
                                             'samples=14\n'
                                             '   iops        : min=402198, '
                                             'max=407152, avg=404296.71, '
                                             'stdev=1609.70, samples=14\n'
                                             '  lat (nsec)   : 750=0.01%, '
                                             '1000=0.01%\n'
                                             '  lat (usec)   : 4=0.01%, '
                                             '10=96.73%, 20=2.70%, 50=0.52%, '
                                             '100=0.04%\n'
                                             '  lat (usec)   : 250=0.01%\n'
                                             '  cpu          : usr=39.81%, '
                                             'sys=60.16%, ctx=12, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.1%, '
                                             '2=49.2%, 4=50.8%, 8=0.0%, '
                                             '16=0.0%, 32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=2829969,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=4\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=197MiB/s (207MB/s), '
                                             '197MiB/s-197MiB/s '
                                             '(207MB/s-207MB/s), io=1382MiB '
                                             '(1449MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_285.output'),
                                   'state': {'begin': 1666783422.153844,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_4_libaio-bdev_xnvme_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8044538497924805,
                                             'end': 1666783422.958298,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_285.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_285.state')},
                       'cmd_286': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_512_4_libaio-bdev_xnvme_2.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_286.output'),
                                   'state': {'begin': 1666783425.066735,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_4_libaio-bdev_xnvme_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7843372821807861,
                                             'end': 1666783425.8510723,
                                             'err': 1,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_286.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_286.state')},
                       'cmd_287': {'output': 'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_287.output'),
                                   'state': {'begin': 1666783425.8533318,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="libaio-bdev_xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="4" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="spdk_bdev" '
                                                    '--spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" '
                                                    '--filename="Nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_4_libaio-bdev_xnvme_2.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.602382898330688,
                                             'end': 1666783437.4557147,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_287.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_287.state')},
                       'cmd_288': {'output': 'libaio-bdev_xnvme: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=spdk_bdev, iodepth=4\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666790527,\n'
                                             '  "timestamp_ms" : '
                                             '1666790527394,\n'
                                             '  "time" : "Wed Oct 26 15:22:07 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"libaio-bdev_xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"libaio-bdev_xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "4",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"spdk_bdev",\n'
                                             '        "spdk_conf" : '
                                             '"/tmp/spdk_bdev_xnvme_libaio.conf",\n'
                                             '        "filename" : "Nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '1440870912,\n'
                                             '        "io_kbytes" : 1407100,\n'
                                             '        "bw_bytes" : 205809300,\n'
                                             '        "bw" : 200985,\n'
                                             '        "iops" : 401970.718469,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 2814197,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1673,\n'
                                             '          "max" : 45495,\n'
                                             '          "mean" : 1855.646163,\n'
                                             '          "stddev" : '
                                             '123.276425,\n'
                                             '          "N" : 2814198\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 682,\n'
                                             '          "max" : 118602,\n'
                                             '          "mean" : 6813.198220,\n'
                                             '          "stddev" : '
                                             '2388.471879,\n'
                                             '          "N" : 2814197,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 5216,\n'
                                             '            "5.000000" : 5280,\n'
                                             '            "10.000000" : 5280,\n'
                                             '            "20.000000" : 5344,\n'
                                             '            "30.000000" : 5344,\n'
                                             '            "40.000000" : 5472,\n'
                                             '            "50.000000" : 7264,\n'
                                             '            "60.000000" : 7328,\n'
                                             '            "70.000000" : 7392,\n'
                                             '            "80.000000" : 7456,\n'
                                             '            "90.000000" : 7904,\n'
                                             '            "95.000000" : 8096,\n'
                                             '            "99.000000" : '
                                             '17536,\n'
                                             '            "99.500000" : '
                                             '21376,\n'
                                             '            "99.900000" : '
                                             '26496,\n'
                                             '            "99.950000" : '
                                             '32128,\n'
                                             '            "99.990000" : 65280\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 5480,\n'
                                             '          "max" : 120416,\n'
                                             '          "mean" : 8668.844394,\n'
                                             '          "stddev" : '
                                             '2391.963514,\n'
                                             '          "N" : 2814197\n'
                                             '        },\n'
                                             '        "bw_min" : 200134,\n'
                                             '        "bw_max" : 201911,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '201021.500000,\n'
                                             '        "bw_dev" : 640.262897,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 400268,\n'
                                             '        "iops_max" : 403823,\n'
                                             '        "iops_mean" : '
                                             '402043.071429,\n'
                                             '        "iops_stddev" : '
                                             '1280.632684,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 40.957143,\n'
                                             '      "sys_cpu" : 59.014286,\n'
                                             '      "ctx" : 20,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.100000,\n'
                                             '        "2" : 47.461923,\n'
                                             '        "4" : 52.534986,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.010000,\n'
                                             '        "1000" : 0.010000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.010000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 96.760248,\n'
                                             '        "20" : 2.542715,\n'
                                             '        "50" : 0.653970,\n'
                                             '        "100" : 0.034681,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 4,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'libaio-bdev_xnvme: (groupid=0, '
                                             'jobs=1): err= 0: pid=9890: Wed '
                                             'Oct 26 15:22:07 2022\n'
                                             '  read: IOPS=402k, BW=196MiB/s '
                                             '(206MB/s)(1374MiB/7001msec)\n'
                                             '    slat (nsec): min=1673, '
                                             'max=45495, avg=1855.65, '
                                             'stdev=123.28\n'
                                             '    clat (nsec): min=682, '
                                             'max=118602, avg=6813.20, '
                                             'stdev=2388.47\n'
                                             '     lat (usec): min=5, max=120, '
                                             'avg= 8.67, stdev= 2.39\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 5216],  '
                                             '5.00th=[ 5280], 10.00th=[ 5280], '
                                             '20.00th=[ 5344],\n'
                                             '     | 30.00th=[ 5344], '
                                             '40.00th=[ 5472], 50.00th=[ '
                                             '7264], 60.00th=[ 7328],\n'
                                             '     | 70.00th=[ 7392], '
                                             '80.00th=[ 7456], 90.00th=[ '
                                             '7904], 95.00th=[ 8096],\n'
                                             '     | 99.00th=[17536], '
                                             '99.50th=[21376], '
                                             '99.90th=[26496], '
                                             '99.95th=[32128],\n'
                                             '     | 99.99th=[65280]\n'
                                             '   bw (  KiB/s): min=200134, '
                                             'max=201911, per=100.00%, '
                                             'avg=201021.50, stdev=640.26, '
                                             'samples=14\n'
                                             '   iops        : min=400268, '
                                             'max=403823, avg=402043.07, '
                                             'stdev=1280.63, samples=14\n'
                                             '  lat (nsec)   : 750=0.01%, '
                                             '1000=0.01%\n'
                                             '  lat (usec)   : 2=0.01%, '
                                             '4=0.01%, 10=96.76%, 20=2.54%, '
                                             '50=0.65%\n'
                                             '  lat (usec)   : 100=0.03%, '
                                             '250=0.01%\n'
                                             '  cpu          : usr=40.96%, '
                                             'sys=59.01%, ctx=20, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.1%, '
                                             '2=47.5%, 4=52.5%, 8=0.0%, '
                                             '16=0.0%, 32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=2814197,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=4\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=196MiB/s (206MB/s), '
                                             '196MiB/s-196MiB/s '
                                             '(206MB/s-206MB/s), io=1374MiB '
                                             '(1441MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_288.output'),
                                   'state': {'begin': 1666783437.45846,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_4_libaio-bdev_xnvme_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.870962381362915,
                                             'end': 1666783438.3294225,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_288.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_288.state')},
                       'cmd_289': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_512_4_libaio-xnvme_0.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_289.output'),
                                   'state': {'begin': 1666783439.4320314,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_4_libaio-xnvme_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8543522357940674,
                                             'end': 1666783440.2863836,
                                             'err': 1,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_289.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_289.state')},
                       'cmd_29': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_29.output'),
                                  'state': {'begin': 1666782176.0607202,
                                            'cmd': '/opt/fio/bin/fio '
                                                   '--name="io_uring-reference" '
                                                   '--rw="randread" '
                                                   '--size="12G" --iodepth="1" '
                                                   '--bs="512" --direct="1" '
                                                   '--thread="1" '
                                                   '--time_based="1" '
                                                   '--runtime="7" '
                                                   '--ramp_time="3" '
                                                   '--norandommap="1" '
                                                   '--allow_file_create="0" '
                                                   '--ioengine="io_uring" '
                                                   '--filename="/dev/nvme0n1" '
                                                   '--output-format="normal,json" '
                                                   '--output="/tmp/fio-output_512_1_io_uring-reference_0.txt" '
                                                   '--cpus_allowed="1"',
                                            'cwd': 'None',
                                            'elapsed': 11.042643070220947,
                                            'end': 1666782187.1033633,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_29.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_29.state')},
                       'cmd_290': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_290.output'),
                                   'state': {'begin': 1666783440.2887886,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="libaio-xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="4" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
                                                    '--xnvme_be="linux" '
                                                    '--xnvme_async="libaio" '
                                                    '--xnvme_sync="nvme" '
                                                    '--xnvme_admin="nvme" '
                                                    '--xnvme_dev_nsid="1" '
                                                    '--filename="/dev/nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_4_libaio-xnvme_0.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.134579181671143,
                                             'end': 1666783451.4233677,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_290.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_290.state')},
                       'cmd_291': {'output': 'libaio-xnvme: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=xnvme, iodepth=4\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666790541,\n'
                                             '  "timestamp_ms" : '
                                             '1666790541496,\n'
                                             '  "time" : "Wed Oct 26 15:22:21 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"libaio-xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"libaio-xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "4",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
                                             '        "xnvme_be" : "linux",\n'
                                             '        "xnvme_async" : '
                                             '"libaio",\n'
                                             '        "xnvme_sync" : "nvme",\n'
                                             '        "xnvme_admin" : "nvme",\n'
                                             '        "xnvme_dev_nsid" : "1",\n'
                                             '        "filename" : '
                                             '"/dev/nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '1465430528,\n'
                                             '        "io_kbytes" : 1431084,\n'
                                             '        "bw_bytes" : 209317315,\n'
                                             '        "bw" : 204411,\n'
                                             '        "iops" : 408822.453935,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 2862166,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1502,\n'
                                             '          "max" : 21721,\n'
                                             '          "mean" : 1667.778486,\n'
                                             '          "stddev" : '
                                             '106.823115,\n'
                                             '          "N" : 2862166\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 511,\n'
                                             '          "max" : 149921,\n'
                                             '          "mean" : 7891.621895,\n'
                                             '          "stddev" : '
                                             '2696.853545,\n'
                                             '          "N" : 2862169,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 7264,\n'
                                             '            "5.000000" : 7328,\n'
                                             '            "10.000000" : 7328,\n'
                                             '            "20.000000" : 7328,\n'
                                             '            "30.000000" : 7392,\n'
                                             '            "40.000000" : 7392,\n'
                                             '            "50.000000" : 7392,\n'
                                             '            "60.000000" : 7456,\n'
                                             '            "70.000000" : 7520,\n'
                                             '            "80.000000" : 7648,\n'
                                             '            "90.000000" : 7776,\n'
                                             '            "95.000000" : 7840,\n'
                                             '            "99.000000" : '
                                             '22656,\n'
                                             '            "99.500000" : '
                                             '27264,\n'
                                             '            "99.900000" : '
                                             '33536,\n'
                                             '            "99.950000" : '
                                             '38656,\n'
                                             '            "99.990000" : 72192\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 6402,\n'
                                             '          "max" : 151595,\n'
                                             '          "mean" : 9559.400362,\n'
                                             '          "stddev" : '
                                             '2700.206986,\n'
                                             '          "N" : 2862169\n'
                                             '        },\n'
                                             '        "bw_min" : 203864,\n'
                                             '        "bw_max" : 205193,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '204511.357143,\n'
                                             '        "bw_dev" : 307.759798,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 407728,\n'
                                             '        "iops_max" : 410386,\n'
                                             '        "iops_mean" : '
                                             '409022.714286,\n'
                                             '        "iops_stddev" : '
                                             '615.561711,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 27.228571,\n'
                                             '      "sys_cpu" : 69.314286,\n'
                                             '      "ctx" : 25260,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.010000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 96.581889,\n'
                                             '        "20" : 2.139429,\n'
                                             '        "50" : 1.239236,\n'
                                             '        "100" : 0.038922,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 4,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'libaio-xnvme: (groupid=0, '
                                             'jobs=1): err= 0: pid=9937: Wed '
                                             'Oct 26 15:22:21 2022\n'
                                             '  read: IOPS=409k, BW=200MiB/s '
                                             '(209MB/s)(1398MiB/7001msec)\n'
                                             '    slat (nsec): min=1502, '
                                             'max=21721, avg=1667.78, '
                                             'stdev=106.82\n'
                                             '    clat (nsec): min=511, '
                                             'max=149921, avg=7891.62, '
                                             'stdev=2696.85\n'
                                             '     lat (usec): min=6, max=151, '
                                             'avg= 9.56, stdev= 2.70\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 7264],  '
                                             '5.00th=[ 7328], 10.00th=[ 7328], '
                                             '20.00th=[ 7328],\n'
                                             '     | 30.00th=[ 7392], '
                                             '40.00th=[ 7392], 50.00th=[ '
                                             '7392], 60.00th=[ 7456],\n'
                                             '     | 70.00th=[ 7520], '
                                             '80.00th=[ 7648], 90.00th=[ '
                                             '7776], 95.00th=[ 7840],\n'
                                             '     | 99.00th=[22656], '
                                             '99.50th=[27264], '
                                             '99.90th=[33536], '
                                             '99.95th=[38656],\n'
                                             '     | 99.99th=[72192]\n'
                                             '   bw (  KiB/s): min=203864, '
                                             'max=205193, per=100.00%, '
                                             'avg=204511.36, stdev=307.76, '
                                             'samples=14\n'
                                             '   iops        : min=407728, '
                                             'max=410386, avg=409022.71, '
                                             'stdev=615.56, samples=14\n'
                                             '  lat (nsec)   : 750=0.01%\n'
                                             '  lat (usec)   : 10=96.58%, '
                                             '20=2.14%, 50=1.24%, 100=0.04%, '
                                             '250=0.01%\n'
                                             '  cpu          : usr=27.23%, '
                                             'sys=69.31%, ctx=25260, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, 2=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=2862166,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=4\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=200MiB/s (209MB/s), '
                                             '200MiB/s-200MiB/s '
                                             '(209MB/s-209MB/s), io=1398MiB '
                                             '(1465MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_291.output'),
                                   'state': {'begin': 1666783451.4265172,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_4_libaio-xnvme_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8377313613891602,
                                             'end': 1666783452.2642486,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_291.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_291.state')},
                       'cmd_292': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_512_4_libaio-xnvme_1.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_292.output'),
                                   'state': {'begin': 1666783453.292761,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_4_libaio-xnvme_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.813385009765625,
                                             'end': 1666783454.106146,
                                             'err': 1,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_292.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_292.state')},
                       'cmd_293': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_293.output'),
                                   'state': {'begin': 1666783454.108579,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="libaio-xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="4" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
                                                    '--xnvme_be="linux" '
                                                    '--xnvme_async="libaio" '
                                                    '--xnvme_sync="nvme" '
                                                    '--xnvme_admin="nvme" '
                                                    '--xnvme_dev_nsid="1" '
                                                    '--filename="/dev/nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_4_libaio-xnvme_1.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.093466997146606,
                                             'end': 1666783465.202046,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_293.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_293.state')},
                       'cmd_294': {'output': 'libaio-xnvme: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=xnvme, iodepth=4\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666790555,\n'
                                             '  "timestamp_ms" : '
                                             '1666790555276,\n'
                                             '  "time" : "Wed Oct 26 15:22:35 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"libaio-xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"libaio-xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "4",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
                                             '        "xnvme_be" : "linux",\n'
                                             '        "xnvme_async" : '
                                             '"libaio",\n'
                                             '        "xnvme_sync" : "nvme",\n'
                                             '        "xnvme_admin" : "nvme",\n'
                                             '        "xnvme_dev_nsid" : "1",\n'
                                             '        "filename" : '
                                             '"/dev/nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '1449910272,\n'
                                             '        "io_kbytes" : 1415928,\n'
                                             '        "bw_bytes" : 207100453,\n'
                                             '        "bw" : 202246,\n'
                                             '        "iops" : 404492.501071,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 2831852,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1533,\n'
                                             '          "max" : 21871,\n'
                                             '          "mean" : 1679.587689,\n'
                                             '          "stddev" : '
                                             '109.338587,\n'
                                             '          "N" : 2831853\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 4919,\n'
                                             '          "max" : 90730,\n'
                                             '          "mean" : 7972.974219,\n'
                                             '          "stddev" : '
                                             '2662.447268,\n'
                                             '          "N" : 2831855,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 7392,\n'
                                             '            "5.000000" : 7392,\n'
                                             '            "10.000000" : 7456,\n'
                                             '            "20.000000" : 7456,\n'
                                             '            "30.000000" : 7456,\n'
                                             '            "40.000000" : 7520,\n'
                                             '            "50.000000" : 7520,\n'
                                             '            "60.000000" : 7520,\n'
                                             '            "70.000000" : 7584,\n'
                                             '            "80.000000" : 7648,\n'
                                             '            "90.000000" : 7840,\n'
                                             '            "95.000000" : 7968,\n'
                                             '            "99.000000" : '
                                             '22400,\n'
                                             '            "99.500000" : '
                                             '27264,\n'
                                             '            "99.900000" : '
                                             '33024,\n'
                                             '            "99.950000" : '
                                             '36608,\n'
                                             '            "99.990000" : 71168\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 6472,\n'
                                             '          "max" : 92383,\n'
                                             '          "mean" : 9652.561871,\n'
                                             '          "stddev" : '
                                             '2665.381488,\n'
                                             '          "N" : 2831855\n'
                                             '        },\n'
                                             '        "bw_min" : 201817,\n'
                                             '        "bw_max" : 203505,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '202362.923077,\n'
                                             '        "bw_dev" : 461.921072,\n'
                                             '        "bw_samples" : 13,\n'
                                             '        "iops_min" : 403634,\n'
                                             '        "iops_max" : 407010,\n'
                                             '        "iops_mean" : '
                                             '404726.153846,\n'
                                             '        "iops_stddev" : '
                                             '923.775843,\n'
                                             '        "iops_samples" : 13\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 32.457143,\n'
                                             '      "sys_cpu" : 64.142857,\n'
                                             '      "ctx" : 25084,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 96.526902,\n'
                                             '        "20" : 2.143403,\n'
                                             '        "50" : 1.292864,\n'
                                             '        "100" : 0.036937,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 4,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'libaio-xnvme: (groupid=0, '
                                             'jobs=1): err= 0: pid=9985: Wed '
                                             'Oct 26 15:22:35 2022\n'
                                             '  read: IOPS=404k, BW=198MiB/s '
                                             '(207MB/s)(1383MiB/7001msec)\n'
                                             '    slat (nsec): min=1533, '
                                             'max=21871, avg=1679.59, '
                                             'stdev=109.34\n'
                                             '    clat (nsec): min=4919, '
                                             'max=90730, avg=7972.97, '
                                             'stdev=2662.45\n'
                                             '     lat (nsec): min=6472, '
                                             'max=92383, avg=9652.56, '
                                             'stdev=2665.38\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 7392],  '
                                             '5.00th=[ 7392], 10.00th=[ 7456], '
                                             '20.00th=[ 7456],\n'
                                             '     | 30.00th=[ 7456], '
                                             '40.00th=[ 7520], 50.00th=[ '
                                             '7520], 60.00th=[ 7520],\n'
                                             '     | 70.00th=[ 7584], '
                                             '80.00th=[ 7648], 90.00th=[ '
                                             '7840], 95.00th=[ 7968],\n'
                                             '     | 99.00th=[22400], '
                                             '99.50th=[27264], '
                                             '99.90th=[33024], '
                                             '99.95th=[36608],\n'
                                             '     | 99.99th=[71168]\n'
                                             '   bw (  KiB/s): min=201817, '
                                             'max=203505, per=100.00%, '
                                             'avg=202362.92, stdev=461.92, '
                                             'samples=13\n'
                                             '   iops        : min=403634, '
                                             'max=407010, avg=404726.15, '
                                             'stdev=923.78, samples=13\n'
                                             '  lat (usec)   : 10=96.53%, '
                                             '20=2.14%, 50=1.29%, 100=0.04%\n'
                                             '  cpu          : usr=32.46%, '
                                             'sys=64.14%, ctx=25084, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, 2=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=2831852,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=4\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=198MiB/s (207MB/s), '
                                             '198MiB/s-198MiB/s '
                                             '(207MB/s-207MB/s), io=1383MiB '
                                             '(1450MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_294.output'),
                                   'state': {'begin': 1666783465.2048554,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_4_libaio-xnvme_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8970766067504883,
                                             'end': 1666783466.101932,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_294.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_294.state')},
                       'cmd_295': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_512_4_libaio-xnvme_2.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_295.output'),
                                   'state': {'begin': 1666783467.3496056,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_4_libaio-xnvme_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 1.1307165622711182,
                                             'end': 1666783468.4803221,
                                             'err': 1,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_295.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_295.state')},
                       'cmd_296': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_296.output'),
                                   'state': {'begin': 1666783468.4822228,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="libaio-xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="4" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
                                                    '--xnvme_be="linux" '
                                                    '--xnvme_async="libaio" '
                                                    '--xnvme_sync="nvme" '
                                                    '--xnvme_admin="nvme" '
                                                    '--xnvme_dev_nsid="1" '
                                                    '--filename="/dev/nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_4_libaio-xnvme_2.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.477175951004028,
                                             'end': 1666783479.9593987,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_296.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_296.state')},
                       'cmd_297': {'output': 'libaio-xnvme: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=xnvme, iodepth=4\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666790570,\n'
                                             '  "timestamp_ms" : '
                                             '1666790570033,\n'
                                             '  "time" : "Wed Oct 26 15:22:50 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"libaio-xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"libaio-xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "4",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
                                             '        "xnvme_be" : "linux",\n'
                                             '        "xnvme_async" : '
                                             '"libaio",\n'
                                             '        "xnvme_sync" : "nvme",\n'
                                             '        "xnvme_admin" : "nvme",\n'
                                             '        "xnvme_dev_nsid" : "1",\n'
                                             '        "filename" : '
                                             '"/dev/nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '1468516864,\n'
                                             '        "io_kbytes" : 1434098,\n'
                                             '        "bw_bytes" : 209758157,\n'
                                             '        "bw" : 204841,\n'
                                             '        "iops" : 409683.330953,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 2868193,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1513,\n'
                                             '          "max" : 21661,\n'
                                             '          "mean" : 1664.947021,\n'
                                             '          "stddev" : '
                                             '111.275646,\n'
                                             '          "N" : 2868194\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 4869,\n'
                                             '          "max" : 125125,\n'
                                             '          "mean" : 7874.002652,\n'
                                             '          "stddev" : '
                                             '2616.785369,\n'
                                             '          "N" : 2868196,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 7264,\n'
                                             '            "5.000000" : 7328,\n'
                                             '            "10.000000" : 7328,\n'
                                             '            "20.000000" : 7392,\n'
                                             '            "30.000000" : 7392,\n'
                                             '            "40.000000" : 7392,\n'
                                             '            "50.000000" : 7392,\n'
                                             '            "60.000000" : 7456,\n'
                                             '            "70.000000" : 7456,\n'
                                             '            "80.000000" : 7520,\n'
                                             '            "90.000000" : 7712,\n'
                                             '            "95.000000" : 7840,\n'
                                             '            "99.000000" : '
                                             '21888,\n'
                                             '            "99.500000" : '
                                             '25984,\n'
                                             '            "99.900000" : '
                                             '33024,\n'
                                             '            "99.950000" : '
                                             '36608,\n'
                                             '            "99.990000" : 70144\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 6412,\n'
                                             '          "max" : 126818,\n'
                                             '          "mean" : 9538.949647,\n'
                                             '          "stddev" : '
                                             '2618.618149,\n'
                                             '          "N" : 2868196\n'
                                             '        },\n'
                                             '        "bw_min" : 203662,\n'
                                             '        "bw_max" : 205924,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '204950.642857,\n'
                                             '        "bw_dev" : 680.498189,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 407324,\n'
                                             '        "iops_max" : 411848,\n'
                                             '        "iops_mean" : '
                                             '409901.571429,\n'
                                             '        "iops_stddev" : '
                                             '1360.938913,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 32.400000,\n'
                                             '      "sys_cpu" : 64.171429,\n'
                                             '      "ctx" : 25312,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 96.547025,\n'
                                             '        "20" : 2.215506,\n'
                                             '        "50" : 1.198664,\n'
                                             '        "100" : 0.038526,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 4,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'libaio-xnvme: (groupid=0, '
                                             'jobs=1): err= 0: pid=10034: Wed '
                                             'Oct 26 15:22:50 2022\n'
                                             '  read: IOPS=410k, BW=200MiB/s '
                                             '(210MB/s)(1400MiB/7001msec)\n'
                                             '    slat (nsec): min=1513, '
                                             'max=21661, avg=1664.95, '
                                             'stdev=111.28\n'
                                             '    clat (usec): min=4, max=125, '
                                             'avg= 7.87, stdev= 2.62\n'
                                             '     lat (usec): min=6, max=126, '
                                             'avg= 9.54, stdev= 2.62\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 7264],  '
                                             '5.00th=[ 7328], 10.00th=[ 7328], '
                                             '20.00th=[ 7392],\n'
                                             '     | 30.00th=[ 7392], '
                                             '40.00th=[ 7392], 50.00th=[ '
                                             '7392], 60.00th=[ 7456],\n'
                                             '     | 70.00th=[ 7456], '
                                             '80.00th=[ 7520], 90.00th=[ '
                                             '7712], 95.00th=[ 7840],\n'
                                             '     | 99.00th=[21888], '
                                             '99.50th=[25984], '
                                             '99.90th=[33024], '
                                             '99.95th=[36608],\n'
                                             '     | 99.99th=[70144]\n'
                                             '   bw (  KiB/s): min=203662, '
                                             'max=205924, per=100.00%, '
                                             'avg=204950.64, stdev=680.50, '
                                             'samples=14\n'
                                             '   iops        : min=407324, '
                                             'max=411848, avg=409901.57, '
                                             'stdev=1360.94, samples=14\n'
                                             '  lat (usec)   : 10=96.55%, '
                                             '20=2.22%, 50=1.20%, 100=0.04%, '
                                             '250=0.01%\n'
                                             '  cpu          : usr=32.40%, '
                                             'sys=64.17%, ctx=25312, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, 2=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=2868193,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=4\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=200MiB/s (210MB/s), '
                                             '200MiB/s-200MiB/s '
                                             '(210MB/s-210MB/s), io=1400MiB '
                                             '(1469MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_297.output'),
                                   'state': {'begin': 1666783479.962116,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_4_libaio-xnvme_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8609395027160645,
                                             'end': 1666783480.8230555,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_297.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_297.state')},
                       'cmd_298': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_512_8_io_uring_cmd-reference_0.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_298.output'),
                                   'state': {'begin': 1666783481.8781722,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_8_io_uring_cmd-reference_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8223638534545898,
                                             'end': 1666783482.700536,
                                             'err': 1,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_298.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_298.state')},
                       'cmd_299': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_299.output'),
                                   'state': {'begin': 1666783482.7027597,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring_cmd-reference" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="8" --bs="512" '
                                                    '--direct="0" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="io_uring_cmd" '
                                                    '--filename="/dev/ng0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_8_io_uring_cmd-reference_0.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.055232286453247,
                                             'end': 1666783493.757992,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_299.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_299.state')},
                       'cmd_30': {'output': 'io_uring-reference: (g=0): '
                                            'rw=randread, bs=(R) 512B-512B, '
                                            '(W) 512B-512B, (T) 512B-512B, '
                                            'ioengine=io_uring, iodepth=1\n'
                                            'fio-3.32-dirty\n'
                                            'Starting 1 thread\n'
                                            '{\n'
                                            '  "fio version" : '
                                            '"fio-3.32-dirty",\n'
                                            '  "timestamp" : 1666789277,\n'
                                            '  "timestamp_ms" : '
                                            '1666789277197,\n'
                                            '  "time" : "Wed Oct 26 15:01:17 '
                                            '2022",\n'
                                            '  "jobs" : [\n'
                                            '    {\n'
                                            '      "jobname" : '
                                            '"io_uring-reference",\n'
                                            '      "groupid" : 0,\n'
                                            '      "error" : 0,\n'
                                            '      "eta" : 0,\n'
                                            '      "elapsed" : 11,\n'
                                            '      "job options" : {\n'
                                            '        "name" : '
                                            '"io_uring-reference",\n'
                                            '        "rw" : "randread",\n'
                                            '        "size" : "12G",\n'
                                            '        "iodepth" : "1",\n'
                                            '        "bs" : "512",\n'
                                            '        "direct" : "1",\n'
                                            '        "thread" : "1",\n'
                                            '        "time_based" : "1",\n'
                                            '        "runtime" : "7",\n'
                                            '        "ramp_time" : "3",\n'
                                            '        "norandommap" : "1",\n'
                                            '        "allow_file_create" : '
                                            '"0",\n'
                                            '        "ioengine" : "io_uring",\n'
                                            '        "filename" : '
                                            '"/dev/nvme0n1",\n'
                                            '        "cpus_allowed" : "1"\n'
                                            '      },\n'
                                            '      "read" : {\n'
                                            '        "io_bytes" : 392608768,\n'
                                            '        "io_kbytes" : 383407,\n'
                                            '        "bw_bytes" : 56078955,\n'
                                            '        "bw" : 54764,\n'
                                            '        "iops" : 109529.067276,\n'
                                            '        "runtime" : 7001,\n'
                                            '        "total_ios" : 766813,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 1443,\n'
                                            '          "max" : 17152,\n'
                                            '          "mean" : 1576.314540,\n'
                                            '          "stddev" : 85.440023,\n'
                                            '          "N" : 766814\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 161,\n'
                                            '          "max" : 78006,\n'
                                            '          "mean" : 7235.024115,\n'
                                            '          "stddev" : 971.994924,\n'
                                            '          "N" : 766813,\n'
                                            '          "percentile" : {\n'
                                            '            "1.000000" : 6752,\n'
                                            '            "5.000000" : 6944,\n'
                                            '            "10.000000" : 7008,\n'
                                            '            "20.000000" : 7008,\n'
                                            '            "30.000000" : 7072,\n'
                                            '            "40.000000" : 7072,\n'
                                            '            "50.000000" : 7136,\n'
                                            '            "60.000000" : 7200,\n'
                                            '            "70.000000" : 7200,\n'
                                            '            "80.000000" : 7264,\n'
                                            '            "90.000000" : 7328,\n'
                                            '            "95.000000" : 7456,\n'
                                            '            "99.000000" : 10304,\n'
                                            '            "99.500000" : 13504,\n'
                                            '            "99.900000" : 15552,\n'
                                            '            "99.950000" : 15808,\n'
                                            '            "99.990000" : 58112\n'
                                            '          }\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 7444,\n'
                                            '          "max" : 79509,\n'
                                            '          "mean" : 8811.338725,\n'
                                            '          "stddev" : 977.342601,\n'
                                            '          "N" : 766813\n'
                                            '        },\n'
                                            '        "bw_min" : 54549,\n'
                                            '        "bw_max" : 55065,\n'
                                            '        "bw_agg" : 100.000000,\n'
                                            '        "bw_mean" : '
                                            '54801.642857,\n'
                                            '        "bw_dev" : 147.506249,\n'
                                            '        "bw_samples" : 14,\n'
                                            '        "iops_min" : 109098,\n'
                                            '        "iops_max" : 110130,\n'
                                            '        "iops_mean" : '
                                            '109603.500000,\n'
                                            '        "iops_stddev" : '
                                            '295.184883,\n'
                                            '        "iops_samples" : 14\n'
                                            '      },\n'
                                            '      "write" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "trim" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "sync" : {\n'
                                            '        "total_ios" : 0,\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        }\n'
                                            '      },\n'
                                            '      "job_runtime" : 7000,\n'
                                            '      "usr_cpu" : 9.400000,\n'
                                            '      "sys_cpu" : 34.500000,\n'
                                            '      "ctx" : 766814,\n'
                                            '      "majf" : 0,\n'
                                            '      "minf" : 0,\n'
                                            '      "iodepth_level" : {\n'
                                            '        "1" : 100.000000,\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_submit" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_complete" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ns" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.010000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_us" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.010000,\n'
                                            '        "10" : 98.745718,\n'
                                            '        "20" : 1.234330,\n'
                                            '        "50" : 0.010000,\n'
                                            '        "100" : 0.013954,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ms" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000,\n'
                                            '        "2000" : 0.000000,\n'
                                            '        ">=2000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_depth" : 1,\n'
                                            '      "latency_target" : 0,\n'
                                            '      "latency_percentile" : '
                                            '100.000000,\n'
                                            '      "latency_window" : 0\n'
                                            '    }\n'
                                            '  ],\n'
                                            '  "disk_util" : [\n'
                                            '    {\n'
                                            '      "name" : "nvme0n1",\n'
                                            '      "read_ios" : 1099741,\n'
                                            '      "write_ios" : 0,\n'
                                            '      "read_merges" : 0,\n'
                                            '      "write_merges" : 0,\n'
                                            '      "read_ticks" : 5523,\n'
                                            '      "write_ticks" : 0,\n'
                                            '      "in_queue" : 5523,\n'
                                            '      "util" : 99.108381\n'
                                            '    }\n'
                                            '  ]\n'
                                            '}\n'
                                            '\n'
                                            'io_uring-reference: (groupid=0, '
                                            'jobs=1): err= 0: pid=5391: Wed '
                                            'Oct 26 15:01:17 2022\n'
                                            '  read: IOPS=110k, BW=53.5MiB/s '
                                            '(56.1MB/s)(374MiB/7001msec)\n'
                                            '    slat (nsec): min=1443, '
                                            'max=17152, avg=1576.31, '
                                            'stdev=85.44\n'
                                            '    clat (nsec): min=161, '
                                            'max=78006, avg=7235.02, '
                                            'stdev=971.99\n'
                                            '     lat (nsec): min=7444, '
                                            'max=79509, avg=8811.34, '
                                            'stdev=977.34\n'
                                            '    clat percentiles (nsec):\n'
                                            '     |  1.00th=[ 6752],  5.00th=[ '
                                            '6944], 10.00th=[ 7008], 20.00th=[ '
                                            '7008],\n'
                                            '     | 30.00th=[ 7072], 40.00th=[ '
                                            '7072], 50.00th=[ 7136], 60.00th=[ '
                                            '7200],\n'
                                            '     | 70.00th=[ 7200], 80.00th=[ '
                                            '7264], 90.00th=[ 7328], 95.00th=[ '
                                            '7456],\n'
                                            '     | 99.00th=[10304], '
                                            '99.50th=[13504], 99.90th=[15552], '
                                            '99.95th=[15808],\n'
                                            '     | 99.99th=[58112]\n'
                                            '   bw (  KiB/s): min=54549, '
                                            'max=55065, per=100.00%, '
                                            'avg=54801.64, stdev=147.51, '
                                            'samples=14\n'
                                            '   iops        : min=109098, '
                                            'max=110130, avg=109603.50, '
                                            'stdev=295.18, samples=14\n'
                                            '  lat (nsec)   : 250=0.01%\n'
                                            '  lat (usec)   : 4=0.01%, '
                                            '10=98.75%, 20=1.23%, 50=0.01%, '
                                            '100=0.01%\n'
                                            '  cpu          : usr=9.40%, '
                                            'sys=34.50%, ctx=766814, majf=0, '
                                            'minf=0\n'
                                            '  IO depths    : 1=100.0%, '
                                            '2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, >=64=0.0%\n'
                                            '     submit    : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     complete  : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     issued rwts: '
                                            'total=766813,0,0,0 short=0,0,0,0 '
                                            'dropped=0,0,0,0\n'
                                            '     latency   : target=0, '
                                            'window=0, percentile=100.00%, '
                                            'depth=1\n'
                                            '\n'
                                            'Run status group 0 (all jobs):\n'
                                            '   READ: bw=53.5MiB/s (56.1MB/s), '
                                            '53.5MiB/s-53.5MiB/s '
                                            '(56.1MB/s-56.1MB/s), io=374MiB '
                                            '(393MB), run=7001-7001msec\n'
                                            '\n'
                                            'Disk stats (read/write):\n'
                                            '  nvme0n1: ios=1099741/0, '
                                            'merge=0/0, ticks=5523/0, '
                                            'in_queue=5523, util=99.11%\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_30.output'),
                                  'state': {'begin': 1666782187.1060665,
                                            'cmd': 'cat '
                                                   '/tmp/fio-output_512_1_io_uring-reference_0.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.9112894535064697,
                                            'end': 1666782188.017356,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_30.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_30.state')},
                       'cmd_300': {'output': 'io_uring_cmd-reference: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=io_uring_cmd, '
                                             'iodepth=8\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666790583,\n'
                                             '  "timestamp_ms" : '
                                             '1666790583831,\n'
                                             '  "time" : "Wed Oct 26 15:23:03 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring_cmd-reference",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring_cmd-reference",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "8",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "0",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"io_uring_cmd",\n'
                                             '        "filename" : '
                                             '"/dev/ng0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '1613466624,\n'
                                             '        "io_kbytes" : 1575651,\n'
                                             '        "bw_bytes" : 230462308,\n'
                                             '        "bw" : 225060,\n'
                                             '        "iops" : 450120.554207,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 3151294,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1172,\n'
                                             '          "max" : 51907,\n'
                                             '          "mean" : 1784.391308,\n'
                                             '          "stddev" : '
                                             '386.919507,\n'
                                             '          "N" : 3151295\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 5240,\n'
                                             '          "max" : 113272,\n'
                                             '          "mean" : '
                                             '15718.233979,\n'
                                             '          "stddev" : '
                                             '3820.322070,\n'
                                             '          "N" : 3151301,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 12224,\n'
                                             '            "5.000000" : 12736,\n'
                                             '            "10.000000" : '
                                             '13120,\n'
                                             '            "20.000000" : '
                                             '14272,\n'
                                             '            "30.000000" : '
                                             '14784,\n'
                                             '            "40.000000" : '
                                             '14912,\n'
                                             '            "50.000000" : '
                                             '15040,\n'
                                             '            "60.000000" : '
                                             '15424,\n'
                                             '            "70.000000" : '
                                             '15680,\n'
                                             '            "80.000000" : '
                                             '16320,\n'
                                             '            "90.000000" : '
                                             '17280,\n'
                                             '            "95.000000" : '
                                             '18048,\n'
                                             '            "99.000000" : '
                                             '36608,\n'
                                             '            "99.500000" : '
                                             '39168,\n'
                                             '            "99.900000" : '
                                             '48384,\n'
                                             '            "99.950000" : '
                                             '64256,\n'
                                             '            "99.990000" : 79360\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 6453,\n'
                                             '          "max" : 114505,\n'
                                             '          "mean" : '
                                             '17502.625444,\n'
                                             '          "stddev" : '
                                             '3781.638897,\n'
                                             '          "N" : 3151301\n'
                                             '        },\n'
                                             '        "bw_min" : 224723,\n'
                                             '        "bw_max" : 225795,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '225187.928571,\n'
                                             '        "bw_dev" : 301.062085,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 449446,\n'
                                             '        "iops_max" : 451591,\n'
                                             '        "iops_mean" : '
                                             '450376.071429,\n'
                                             '        "iops_stddev" : '
                                             '602.094097,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 16.871429,\n'
                                             '      "sys_cpu" : 79.900000,\n'
                                             '      "ctx" : 15021,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 100.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 99.999968,\n'
                                             '        "8" : 0.100000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.044109,\n'
                                             '        "20" : 96.321670,\n'
                                             '        "50" : 3.541847,\n'
                                             '        "100" : 0.092248,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 8,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring_cmd-reference: '
                                             '(groupid=0, jobs=1): err= 0: '
                                             'pid=10082: Wed Oct 26 15:23:03 '
                                             '2022\n'
                                             '  read: IOPS=450k, BW=220MiB/s '
                                             '(230MB/s)(1539MiB/7001msec)\n'
                                             '    slat (nsec): min=1172, '
                                             'max=51907, avg=1784.39, '
                                             'stdev=386.92\n'
                                             '    clat (usec): min=5, max=113, '
                                             'avg=15.72, stdev= 3.82\n'
                                             '     lat (usec): min=6, max=114, '
                                             'avg=17.50, stdev= 3.78\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[12224],  '
                                             '5.00th=[12736], 10.00th=[13120], '
                                             '20.00th=[14272],\n'
                                             '     | 30.00th=[14784], '
                                             '40.00th=[14912], '
                                             '50.00th=[15040], '
                                             '60.00th=[15424],\n'
                                             '     | 70.00th=[15680], '
                                             '80.00th=[16320], '
                                             '90.00th=[17280], '
                                             '95.00th=[18048],\n'
                                             '     | 99.00th=[36608], '
                                             '99.50th=[39168], '
                                             '99.90th=[48384], '
                                             '99.95th=[64256],\n'
                                             '     | 99.99th=[79360]\n'
                                             '   bw (  KiB/s): min=224723, '
                                             'max=225795, per=100.00%, '
                                             'avg=225187.93, stdev=301.06, '
                                             'samples=14\n'
                                             '   iops        : min=449446, '
                                             'max=451591, avg=450376.07, '
                                             'stdev=602.09, samples=14\n'
                                             '  lat (usec)   : 10=0.04%, '
                                             '20=96.32%, 50=3.54%, 100=0.09%, '
                                             '250=0.01%\n'
                                             '  cpu          : usr=16.87%, '
                                             'sys=79.90%, ctx=15021, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, 2=0.0%, '
                                             '4=0.0%, 8=100.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.1%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=3151294,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=8\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=220MiB/s (230MB/s), '
                                             '220MiB/s-220MiB/s '
                                             '(230MB/s-230MB/s), io=1539MiB '
                                             '(1613MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_300.output'),
                                   'state': {'begin': 1666783493.7646198,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_8_io_uring_cmd-reference_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8438622951507568,
                                             'end': 1666783494.6084821,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_300.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_300.state')},
                       'cmd_301': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_512_8_io_uring_cmd-reference_1.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_301.output'),
                                   'state': {'begin': 1666783495.661168,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_8_io_uring_cmd-reference_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8514082431793213,
                                             'end': 1666783496.5125763,
                                             'err': 1,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_301.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_301.state')},
                       'cmd_302': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_302.output'),
                                   'state': {'begin': 1666783496.515712,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring_cmd-reference" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="8" --bs="512" '
                                                    '--direct="0" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="io_uring_cmd" '
                                                    '--filename="/dev/ng0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_8_io_uring_cmd-reference_1.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.080080032348633,
                                             'end': 1666783507.595792,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_302.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_302.state')},
                       'cmd_303': {'output': 'io_uring_cmd-reference: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=io_uring_cmd, '
                                             'iodepth=8\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666790597,\n'
                                             '  "timestamp_ms" : '
                                             '1666790597635,\n'
                                             '  "time" : "Wed Oct 26 15:23:17 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring_cmd-reference",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring_cmd-reference",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "8",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "0",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"io_uring_cmd",\n'
                                             '        "filename" : '
                                             '"/dev/ng0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '1596023808,\n'
                                             '        "io_kbytes" : 1558617,\n'
                                             '        "bw_bytes" : 227970833,\n'
                                             '        "bw" : 222627,\n'
                                             '        "iops" : 445254.392230,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 3117226,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1192,\n'
                                             '          "max" : 26660,\n'
                                             '          "mean" : 1791.133760,\n'
                                             '          "stddev" : '
                                             '379.567079,\n'
                                             '          "N" : 3117227\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 1963,\n'
                                             '          "max" : 125385,\n'
                                             '          "mean" : '
                                             '15887.595758,\n'
                                             '          "stddev" : '
                                             '3772.244360,\n'
                                             '          "N" : 3117233,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 12224,\n'
                                             '            "5.000000" : 12864,\n'
                                             '            "10.000000" : '
                                             '13248,\n'
                                             '            "20.000000" : '
                                             '14528,\n'
                                             '            "30.000000" : '
                                             '14912,\n'
                                             '            "40.000000" : '
                                             '15168,\n'
                                             '            "50.000000" : '
                                             '15296,\n'
                                             '            "60.000000" : '
                                             '15680,\n'
                                             '            "70.000000" : '
                                             '15936,\n'
                                             '            "80.000000" : '
                                             '16512,\n'
                                             '            "90.000000" : '
                                             '17536,\n'
                                             '            "95.000000" : '
                                             '18048,\n'
                                             '            "99.000000" : '
                                             '36608,\n'
                                             '            "99.500000" : '
                                             '38656,\n'
                                             '            "99.900000" : '
                                             '47872,\n'
                                             '            "99.950000" : '
                                             '63744,\n'
                                             '            "99.990000" : 79360\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 6492,\n'
                                             '          "max" : 126838,\n'
                                             '          "mean" : '
                                             '17678.729535,\n'
                                             '          "stddev" : '
                                             '3741.390408,\n'
                                             '          "N" : 3117233\n'
                                             '        },\n'
                                             '        "bw_min" : 221913,\n'
                                             '        "bw_max" : 223653,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '222667.071429,\n'
                                             '        "bw_dev" : 489.091224,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 443826,\n'
                                             '        "iops_max" : 447306,\n'
                                             '        "iops_mean" : '
                                             '445334.142857,\n'
                                             '        "iops_stddev" : '
                                             '978.182448,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 17.685714,\n'
                                             '      "sys_cpu" : 79.200000,\n'
                                             '      "ctx" : 14928,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 100.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 99.999968,\n'
                                             '        "8" : 0.100000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.010000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 0.050077,\n'
                                             '        "20" : 96.374341,\n'
                                             '        "50" : 3.485727,\n'
                                             '        "100" : 0.089374,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 8,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring_cmd-reference: '
                                             '(groupid=0, jobs=1): err= 0: '
                                             'pid=10128: Wed Oct 26 15:23:17 '
                                             '2022\n'
                                             '  read: IOPS=445k, BW=217MiB/s '
                                             '(228MB/s)(1522MiB/7001msec)\n'
                                             '    slat (nsec): min=1192, '
                                             'max=26660, avg=1791.13, '
                                             'stdev=379.57\n'
                                             '    clat (nsec): min=1963, '
                                             'max=125385, avg=15887.60, '
                                             'stdev=3772.24\n'
                                             '     lat (usec): min=6, max=126, '
                                             'avg=17.68, stdev= 3.74\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[12224],  '
                                             '5.00th=[12864], 10.00th=[13248], '
                                             '20.00th=[14528],\n'
                                             '     | 30.00th=[14912], '
                                             '40.00th=[15168], '
                                             '50.00th=[15296], '
                                             '60.00th=[15680],\n'
                                             '     | 70.00th=[15936], '
                                             '80.00th=[16512], '
                                             '90.00th=[17536], '
                                             '95.00th=[18048],\n'
                                             '     | 99.00th=[36608], '
                                             '99.50th=[38656], '
                                             '99.90th=[47872], '
                                             '99.95th=[63744],\n'
                                             '     | 99.99th=[79360]\n'
                                             '   bw (  KiB/s): min=221913, '
                                             'max=223653, per=100.00%, '
                                             'avg=222667.07, stdev=489.09, '
                                             'samples=14\n'
                                             '   iops        : min=443826, '
                                             'max=447306, avg=445334.14, '
                                             'stdev=978.18, samples=14\n'
                                             '  lat (usec)   : 2=0.01%, '
                                             '4=0.01%, 10=0.05%, 20=96.37%, '
                                             '50=3.49%\n'
                                             '  lat (usec)   : 100=0.09%, '
                                             '250=0.01%\n'
                                             '  cpu          : usr=17.69%, '
                                             'sys=79.20%, ctx=14928, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, 2=0.0%, '
                                             '4=0.0%, 8=100.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.1%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=3117226,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=8\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=217MiB/s (228MB/s), '
                                             '217MiB/s-217MiB/s '
                                             '(228MB/s-228MB/s), io=1522MiB '
                                             '(1596MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_303.output'),
                                   'state': {'begin': 1666783507.5988343,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_8_io_uring_cmd-reference_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8206968307495117,
                                             'end': 1666783508.419531,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_303.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_303.state')},
                       'cmd_304': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_512_8_io_uring_cmd-reference_2.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_304.output'),
                                   'state': {'begin': 1666783509.4199626,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_8_io_uring_cmd-reference_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8256092071533203,
                                             'end': 1666783510.2455719,
                                             'err': 1,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_304.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_304.state')},
                       'cmd_305': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_305.output'),
                                   'state': {'begin': 1666783510.2464507,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring_cmd-reference" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="8" --bs="512" '
                                                    '--direct="0" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="io_uring_cmd" '
                                                    '--filename="/dev/ng0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_8_io_uring_cmd-reference_2.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.052099466323853,
                                             'end': 1666783521.2985501,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_305.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_305.state')},
                       'cmd_306': {'output': 'io_uring_cmd-reference: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=io_uring_cmd, '
                                             'iodepth=8\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666790611,\n'
                                             '  "timestamp_ms" : '
                                             '1666790611370,\n'
                                             '  "time" : "Wed Oct 26 15:23:31 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring_cmd-reference",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring_cmd-reference",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "8",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "0",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"io_uring_cmd",\n'
                                             '        "filename" : '
                                             '"/dev/ng0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '1604506112,\n'
                                             '        "io_kbytes" : 1566900,\n'
                                             '        "bw_bytes" : 229182418,\n'
                                             '        "bw" : 223810,\n'
                                             '        "iops" : 447620.768462,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 3133793,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1162,\n'
                                             '          "max" : 23243,\n'
                                             '          "mean" : 1790.342289,\n'
                                             '          "stddev" : '
                                             '400.121691,\n'
                                             '          "N" : 3133794\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 1874,\n'
                                             '          "max" : 130384,\n'
                                             '          "mean" : '
                                             '15806.641568,\n'
                                             '          "stddev" : '
                                             '3805.133567,\n'
                                             '          "N" : 3133800,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 12096,\n'
                                             '            "5.000000" : 12736,\n'
                                             '            "10.000000" : '
                                             '13248,\n'
                                             '            "20.000000" : '
                                             '14400,\n'
                                             '            "30.000000" : '
                                             '14784,\n'
                                             '            "40.000000" : '
                                             '15040,\n'
                                             '            "50.000000" : '
                                             '15168,\n'
                                             '            "60.000000" : '
                                             '15552,\n'
                                             '            "70.000000" : '
                                             '15808,\n'
                                             '            "80.000000" : '
                                             '16512,\n'
                                             '            "90.000000" : '
                                             '17280,\n'
                                             '            "95.000000" : '
                                             '18048,\n'
                                             '            "99.000000" : '
                                             '36608,\n'
                                             '            "99.500000" : '
                                             '38656,\n'
                                             '            "99.900000" : '
                                             '48384,\n'
                                             '            "99.950000" : '
                                             '64256,\n'
                                             '            "99.990000" : 79360\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 6532,\n'
                                             '          "max" : 131657,\n'
                                             '          "mean" : '
                                             '17596.983862,\n'
                                             '          "stddev" : '
                                             '3768.535898,\n'
                                             '          "N" : 3133800\n'
                                             '        },\n'
                                             '        "bw_min" : 222819,\n'
                                             '        "bw_max" : 225676,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '223915.384615,\n'
                                             '        "bw_dev" : 891.737867,\n'
                                             '        "bw_samples" : 13,\n'
                                             '        "iops_min" : 445638,\n'
                                             '        "iops_max" : 451352,\n'
                                             '        "iops_mean" : '
                                             '447830.846154,\n'
                                             '        "iops_stddev" : '
                                             '1783.386285,\n'
                                             '        "iops_samples" : 13\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 17.214286,\n'
                                             '      "sys_cpu" : 79.585714,\n'
                                             '      "ctx" : 14974,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 100.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 99.999968,\n'
                                             '        "8" : 0.100000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.010000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.049142,\n'
                                             '        "20" : 96.312360,\n'
                                             '        "50" : 3.546022,\n'
                                             '        "100" : 0.092412,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 8,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring_cmd-reference: '
                                             '(groupid=0, jobs=1): err= 0: '
                                             'pid=10175: Wed Oct 26 15:23:31 '
                                             '2022\n'
                                             '  read: IOPS=448k, BW=219MiB/s '
                                             '(229MB/s)(1530MiB/7001msec)\n'
                                             '    slat (nsec): min=1162, '
                                             'max=23243, avg=1790.34, '
                                             'stdev=400.12\n'
                                             '    clat (nsec): min=1874, '
                                             'max=130384, avg=15806.64, '
                                             'stdev=3805.13\n'
                                             '     lat (usec): min=6, max=131, '
                                             'avg=17.60, stdev= 3.77\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[12096],  '
                                             '5.00th=[12736], 10.00th=[13248], '
                                             '20.00th=[14400],\n'
                                             '     | 30.00th=[14784], '
                                             '40.00th=[15040], '
                                             '50.00th=[15168], '
                                             '60.00th=[15552],\n'
                                             '     | 70.00th=[15808], '
                                             '80.00th=[16512], '
                                             '90.00th=[17280], '
                                             '95.00th=[18048],\n'
                                             '     | 99.00th=[36608], '
                                             '99.50th=[38656], '
                                             '99.90th=[48384], '
                                             '99.95th=[64256],\n'
                                             '     | 99.99th=[79360]\n'
                                             '   bw (  KiB/s): min=222819, '
                                             'max=225676, per=100.00%, '
                                             'avg=223915.38, stdev=891.74, '
                                             'samples=13\n'
                                             '   iops        : min=445638, '
                                             'max=451352, avg=447830.85, '
                                             'stdev=1783.39, samples=13\n'
                                             '  lat (usec)   : 2=0.01%, '
                                             '10=0.05%, 20=96.31%, 50=3.55%, '
                                             '100=0.09%\n'
                                             '  lat (usec)   : 250=0.01%\n'
                                             '  cpu          : usr=17.21%, '
                                             'sys=79.59%, ctx=14974, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, 2=0.0%, '
                                             '4=0.0%, 8=100.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.1%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=3133793,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=8\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=219MiB/s (229MB/s), '
                                             '219MiB/s-219MiB/s '
                                             '(229MB/s-229MB/s), io=1530MiB '
                                             '(1605MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_306.output'),
                                   'state': {'begin': 1666783521.305068,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_8_io_uring_cmd-reference_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.9138171672821045,
                                             'end': 1666783522.2188852,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_306.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_306.state')},
                       'cmd_307': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_512_8_io_uring_cmd-xnvme_0.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_307.output'),
                                   'state': {'begin': 1666783523.3202078,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_8_io_uring_cmd-xnvme_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8594181537628174,
                                             'end': 1666783524.179626,
                                             'err': 1,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_307.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_307.state')},
                       'cmd_308': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_308.output'),
                                   'state': {'begin': 1666783524.1820672,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring_cmd-xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="8" --bs="512" '
                                                    '--direct="0" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
                                                    '--xnvme_be="linux" '
                                                    '--xnvme_async="io_uring_cmd" '
                                                    '--xnvme_sync="nvme" '
                                                    '--xnvme_admin="nvme" '
                                                    '--xnvme_dev_nsid="1" '
                                                    '--filename="/dev/ng0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_8_io_uring_cmd-xnvme_0.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.05566143989563,
                                             'end': 1666783535.2377286,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_308.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_308.state')},
                       'cmd_309': {'output': 'io_uring_cmd-xnvme: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=xnvme, iodepth=8\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666790625,\n'
                                             '  "timestamp_ms" : '
                                             '1666790625309,\n'
                                             '  "time" : "Wed Oct 26 15:23:45 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring_cmd-xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring_cmd-xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "8",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "0",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
                                             '        "xnvme_be" : "linux",\n'
                                             '        "xnvme_async" : '
                                             '"io_uring_cmd",\n'
                                             '        "xnvme_sync" : "nvme",\n'
                                             '        "xnvme_admin" : "nvme",\n'
                                             '        "xnvme_dev_nsid" : "1",\n'
                                             '        "filename" : '
                                             '"/dev/ng0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '1514349056,\n'
                                             '        "io_kbytes" : 1478856,\n'
                                             '        "bw_bytes" : 216335579,\n'
                                             '        "bw" : 211265,\n'
                                             '        "iops" : 422529.285714,\n'
                                             '        "runtime" : 7000,\n'
                                             '        "total_ios" : 2957705,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1232,\n'
                                             '          "max" : 23514,\n'
                                             '          "mean" : 1841.139255,\n'
                                             '          "stddev" : '
                                             '389.694801,\n'
                                             '          "N" : 2957706\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 3677,\n'
                                             '          "max" : 120536,\n'
                                             '          "mean" : '
                                             '16780.929818,\n'
                                             '          "stddev" : '
                                             '3696.313221,\n'
                                             '          "N" : 2957712,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 12608,\n'
                                             '            "5.000000" : 13248,\n'
                                             '            "10.000000" : '
                                             '14400,\n'
                                             '            "20.000000" : '
                                             '15296,\n'
                                             '            "30.000000" : '
                                             '15808,\n'
                                             '            "40.000000" : '
                                             '16192,\n'
                                             '            "50.000000" : '
                                             '16512,\n'
                                             '            "60.000000" : '
                                             '16768,\n'
                                             '            "70.000000" : '
                                             '17024,\n'
                                             '            "80.000000" : '
                                             '17536,\n'
                                             '            "90.000000" : '
                                             '17792,\n'
                                             '            "95.000000" : '
                                             '18560,\n'
                                             '            "99.000000" : '
                                             '36608,\n'
                                             '            "99.500000" : '
                                             '38656,\n'
                                             '            "99.900000" : '
                                             '50432,\n'
                                             '            "99.950000" : '
                                             '66048,\n'
                                             '            "99.990000" : 81408\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 6672,\n'
                                             '          "max" : 121879,\n'
                                             '          "mean" : '
                                             '18622.069282,\n'
                                             '          "stddev" : '
                                             '3671.329577,\n'
                                             '          "N" : 2957712\n'
                                             '        },\n'
                                             '        "bw_min" : 208536,\n'
                                             '        "bw_max" : 213329,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '211349.428571,\n'
                                             '        "bw_dev" : 1221.796830,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 417072,\n'
                                             '        "iops_max" : 426658,\n'
                                             '        "iops_mean" : '
                                             '422699.000000,\n'
                                             '        "iops_stddev" : '
                                             '2443.594893,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 20.442857,\n'
                                             '      "sys_cpu" : 76.714286,\n'
                                             '      "ctx" : 14856,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 100.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 99.999983,\n'
                                             '        "8" : 0.100000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 0.040910,\n'
                                             '        "20" : 96.235764,\n'
                                             '        "50" : 3.622944,\n'
                                             '        "100" : 0.100111,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 8,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring_cmd-xnvme: (groupid=0, '
                                             'jobs=1): err= 0: pid=10221: Wed '
                                             'Oct 26 15:23:45 2022\n'
                                             '  read: IOPS=423k, BW=206MiB/s '
                                             '(216MB/s)(1444MiB/7000msec)\n'
                                             '    slat (nsec): min=1232, '
                                             'max=23514, avg=1841.14, '
                                             'stdev=389.69\n'
                                             '    clat (usec): min=3, max=120, '
                                             'avg=16.78, stdev= 3.70\n'
                                             '     lat (usec): min=6, max=121, '
                                             'avg=18.62, stdev= 3.67\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[12608],  '
                                             '5.00th=[13248], 10.00th=[14400], '
                                             '20.00th=[15296],\n'
                                             '     | 30.00th=[15808], '
                                             '40.00th=[16192], '
                                             '50.00th=[16512], '
                                             '60.00th=[16768],\n'
                                             '     | 70.00th=[17024], '
                                             '80.00th=[17536], '
                                             '90.00th=[17792], '
                                             '95.00th=[18560],\n'
                                             '     | 99.00th=[36608], '
                                             '99.50th=[38656], '
                                             '99.90th=[50432], '
                                             '99.95th=[66048],\n'
                                             '     | 99.99th=[81408]\n'
                                             '   bw (  KiB/s): min=208536, '
                                             'max=213329, per=100.00%, '
                                             'avg=211349.43, stdev=1221.80, '
                                             'samples=14\n'
                                             '   iops        : min=417072, '
                                             'max=426658, avg=422699.00, '
                                             'stdev=2443.59, samples=14\n'
                                             '  lat (usec)   : 4=0.01%, '
                                             '10=0.04%, 20=96.24%, 50=3.62%, '
                                             '100=0.10%\n'
                                             '  lat (usec)   : 250=0.01%\n'
                                             '  cpu          : usr=20.44%, '
                                             'sys=76.71%, ctx=14856, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, 2=0.0%, '
                                             '4=0.0%, 8=100.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.1%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=2957705,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=8\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=206MiB/s (216MB/s), '
                                             '206MiB/s-206MiB/s '
                                             '(216MB/s-216MB/s), io=1444MiB '
                                             '(1514MB), run=7000-7000msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_309.output'),
                                   'state': {'begin': 1666783535.2407825,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_8_io_uring_cmd-xnvme_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8338701725006104,
                                             'end': 1666783536.0746527,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_309.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_309.state')},
                       'cmd_31': {'output': 'rm: cannot remove '
                                            "'/tmp/fio-output_512_1_io_uring-reference_1.txt': "
                                            'No such file or directory\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_31.output'),
                                  'state': {'begin': 1666782189.144373,
                                            'cmd': 'rm '
                                                   '/tmp/fio-output_512_1_io_uring-reference_1.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.8392891883850098,
                                            'end': 1666782189.9836621,
                                            'err': 1,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_31.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_31.state')},
                       'cmd_310': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_512_8_io_uring_cmd-xnvme_1.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_310.output'),
                                   'state': {'begin': 1666783537.0844007,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_8_io_uring_cmd-xnvme_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8432741165161133,
                                             'end': 1666783537.9276748,
                                             'err': 1,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_310.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_310.state')},
                       'cmd_311': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_311.output'),
                                   'state': {'begin': 1666783537.9298892,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring_cmd-xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="8" --bs="512" '
                                                    '--direct="0" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
                                                    '--xnvme_be="linux" '
                                                    '--xnvme_async="io_uring_cmd" '
                                                    '--xnvme_sync="nvme" '
                                                    '--xnvme_admin="nvme" '
                                                    '--xnvme_dev_nsid="1" '
                                                    '--filename="/dev/ng0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_8_io_uring_cmd-xnvme_1.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.05512261390686,
                                             'end': 1666783548.9850118,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_311.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_311.state')},
                       'cmd_312': {'output': 'io_uring_cmd-xnvme: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=xnvme, iodepth=8\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666790639,\n'
                                             '  "timestamp_ms" : '
                                             '1666790639058,\n'
                                             '  "time" : "Wed Oct 26 15:23:59 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring_cmd-xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring_cmd-xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "8",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "0",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
                                             '        "xnvme_be" : "linux",\n'
                                             '        "xnvme_async" : '
                                             '"io_uring_cmd",\n'
                                             '        "xnvme_sync" : "nvme",\n'
                                             '        "xnvme_admin" : "nvme",\n'
                                             '        "xnvme_dev_nsid" : "1",\n'
                                             '        "filename" : '
                                             '"/dev/ng0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '1536173056,\n'
                                             '        "io_kbytes" : 1500169,\n'
                                             '        "bw_bytes" : 219421947,\n'
                                             '        "bw" : 214279,\n'
                                             '        "iops" : 428557.491787,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 3000331,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1222,\n'
                                             '          "max" : 23965,\n'
                                             '          "mean" : 1803.258623,\n'
                                             '          "stddev" : '
                                             '390.548360,\n'
                                             '          "N" : 3000331\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 4839,\n'
                                             '          "max" : 128591,\n'
                                             '          "mean" : '
                                             '16544.098031,\n'
                                             '          "stddev" : '
                                             '3761.336130,\n'
                                             '          "N" : 3000338,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 12608,\n'
                                             '            "5.000000" : 13120,\n'
                                             '            "10.000000" : '
                                             '13760,\n'
                                             '            "20.000000" : '
                                             '15040,\n'
                                             '            "30.000000" : '
                                             '15552,\n'
                                             '            "40.000000" : '
                                             '15936,\n'
                                             '            "50.000000" : '
                                             '16192,\n'
                                             '            "60.000000" : '
                                             '16512,\n'
                                             '            "70.000000" : '
                                             '16768,\n'
                                             '            "80.000000" : '
                                             '17280,\n'
                                             '            "90.000000" : '
                                             '17536,\n'
                                             '            "95.000000" : '
                                             '18560,\n'
                                             '            "99.000000" : '
                                             '37120,\n'
                                             '            "99.500000" : '
                                             '39168,\n'
                                             '            "99.900000" : '
                                             '51456,\n'
                                             '            "99.950000" : '
                                             '67072,\n'
                                             '            "99.990000" : 81408\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 6191,\n'
                                             '          "max" : 129934,\n'
                                             '          "mean" : '
                                             '18347.356140,\n'
                                             '          "stddev" : '
                                             '3737.823304,\n'
                                             '          "N" : 3000338\n'
                                             '        },\n'
                                             '        "bw_min" : 211728,\n'
                                             '        "bw_max" : 216886,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '214318.285714,\n'
                                             '        "bw_dev" : 1642.356301,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 423456,\n'
                                             '        "iops_max" : 433772,\n'
                                             '        "iops_mean" : '
                                             '428636.857143,\n'
                                             '        "iops_stddev" : '
                                             '3284.857914,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 22.057143,\n'
                                             '      "sys_cpu" : 75.014286,\n'
                                             '      "ctx" : 14798,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 100.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 99.999983,\n'
                                             '        "8" : 0.100000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.043895,\n'
                                             '        "20" : 96.254080,\n'
                                             '        "50" : 3.599036,\n'
                                             '        "100" : 0.101522,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 8,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring_cmd-xnvme: (groupid=0, '
                                             'jobs=1): err= 0: pid=10267: Wed '
                                             'Oct 26 15:23:59 2022\n'
                                             '  read: IOPS=429k, BW=209MiB/s '
                                             '(219MB/s)(1465MiB/7001msec)\n'
                                             '    slat (nsec): min=1222, '
                                             'max=23965, avg=1803.26, '
                                             'stdev=390.55\n'
                                             '    clat (usec): min=4, max=128, '
                                             'avg=16.54, stdev= 3.76\n'
                                             '     lat (usec): min=6, max=129, '
                                             'avg=18.35, stdev= 3.74\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[12608],  '
                                             '5.00th=[13120], 10.00th=[13760], '
                                             '20.00th=[15040],\n'
                                             '     | 30.00th=[15552], '
                                             '40.00th=[15936], '
                                             '50.00th=[16192], '
                                             '60.00th=[16512],\n'
                                             '     | 70.00th=[16768], '
                                             '80.00th=[17280], '
                                             '90.00th=[17536], '
                                             '95.00th=[18560],\n'
                                             '     | 99.00th=[37120], '
                                             '99.50th=[39168], '
                                             '99.90th=[51456], '
                                             '99.95th=[67072],\n'
                                             '     | 99.99th=[81408]\n'
                                             '   bw (  KiB/s): min=211728, '
                                             'max=216886, per=100.00%, '
                                             'avg=214318.29, stdev=1642.36, '
                                             'samples=14\n'
                                             '   iops        : min=423456, '
                                             'max=433772, avg=428636.86, '
                                             'stdev=3284.86, samples=14\n'
                                             '  lat (usec)   : 10=0.04%, '
                                             '20=96.25%, 50=3.60%, 100=0.10%, '
                                             '250=0.01%\n'
                                             '  cpu          : usr=22.06%, '
                                             'sys=75.01%, ctx=14798, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, 2=0.0%, '
                                             '4=0.0%, 8=100.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.1%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=3000331,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=8\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=209MiB/s (219MB/s), '
                                             '209MiB/s-209MiB/s '
                                             '(219MB/s-219MB/s), io=1465MiB '
                                             '(1536MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_312.output'),
                                   'state': {'begin': 1666783548.991464,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_8_io_uring_cmd-xnvme_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.795210599899292,
                                             'end': 1666783549.7866745,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_312.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_312.state')},
                       'cmd_313': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_512_8_io_uring_cmd-xnvme_2.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_313.output'),
                                   'state': {'begin': 1666783550.8153117,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_8_io_uring_cmd-xnvme_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.809546709060669,
                                             'end': 1666783551.6248584,
                                             'err': 1,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_313.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_313.state')},
                       'cmd_314': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_314.output'),
                                   'state': {'begin': 1666783551.6267784,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring_cmd-xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="8" --bs="512" '
                                                    '--direct="0" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
                                                    '--xnvme_be="linux" '
                                                    '--xnvme_async="io_uring_cmd" '
                                                    '--xnvme_sync="nvme" '
                                                    '--xnvme_admin="nvme" '
                                                    '--xnvme_dev_nsid="1" '
                                                    '--filename="/dev/ng0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_8_io_uring_cmd-xnvme_2.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.101061820983887,
                                             'end': 1666783562.7278402,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_314.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_314.state')},
                       'cmd_315': {'output': 'io_uring_cmd-xnvme: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=xnvme, iodepth=8\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666790652,\n'
                                             '  "timestamp_ms" : '
                                             '1666790652800,\n'
                                             '  "time" : "Wed Oct 26 15:24:12 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring_cmd-xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring_cmd-xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "8",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "0",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
                                             '        "xnvme_be" : "linux",\n'
                                             '        "xnvme_async" : '
                                             '"io_uring_cmd",\n'
                                             '        "xnvme_sync" : "nvme",\n'
                                             '        "xnvme_admin" : "nvme",\n'
                                             '        "xnvme_dev_nsid" : "1",\n'
                                             '        "filename" : '
                                             '"/dev/ng0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '1528946176,\n'
                                             '        "io_kbytes" : 1493111,\n'
                                             '        "bw_bytes" : 218389683,\n'
                                             '        "bw" : 213271,\n'
                                             '        "iops" : 426541.208399,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 2986215,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1222,\n'
                                             '          "max" : 37791,\n'
                                             '          "mean" : 1814.158787,\n'
                                             '          "stddev" : '
                                             '388.864751,\n'
                                             '          "N" : 2986216\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 121,\n'
                                             '          "max" : 127819,\n'
                                             '          "mean" : '
                                             '16622.123058,\n'
                                             '          "stddev" : '
                                             '3687.178497,\n'
                                             '          "N" : 2986222,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 12608,\n'
                                             '            "5.000000" : 13248,\n'
                                             '            "10.000000" : '
                                             '14144,\n'
                                             '            "20.000000" : '
                                             '15296,\n'
                                             '            "30.000000" : '
                                             '15680,\n'
                                             '            "40.000000" : '
                                             '16064,\n'
                                             '            "50.000000" : '
                                             '16192,\n'
                                             '            "60.000000" : '
                                             '16512,\n'
                                             '            "70.000000" : '
                                             '16768,\n'
                                             '            "80.000000" : '
                                             '17280,\n'
                                             '            "90.000000" : '
                                             '17536,\n'
                                             '            "95.000000" : '
                                             '18560,\n'
                                             '            "99.000000" : '
                                             '36608,\n'
                                             '            "99.500000" : '
                                             '38656,\n'
                                             '            "99.900000" : '
                                             '48896,\n'
                                             '            "99.950000" : '
                                             '65280,\n'
                                             '            "99.990000" : 80384\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 6693,\n'
                                             '          "max" : 129142,\n'
                                             '          "mean" : '
                                             '18436.281854,\n'
                                             '          "stddev" : '
                                             '3660.248095,\n'
                                             '          "N" : 2986222\n'
                                             '        },\n'
                                             '        "bw_min" : 212347,\n'
                                             '        "bw_max" : 214606,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '213309.142857,\n'
                                             '        "bw_dev" : 669.714733,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 424694,\n'
                                             '        "iops_max" : 429212,\n'
                                             '        "iops_mean" : '
                                             '426618.285714,\n'
                                             '        "iops_stddev" : '
                                             '1339.429467,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 21.257143,\n'
                                             '      "sys_cpu" : 75.814286,\n'
                                             '      "ctx" : 14684,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 1,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 100.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 99.999983,\n'
                                             '        "8" : 0.100000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.037707,\n'
                                             '        "20" : 96.299061,\n'
                                             '        "50" : 3.567292,\n'
                                             '        "100" : 0.095405,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 8,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring_cmd-xnvme: (groupid=0, '
                                             'jobs=1): err= 0: pid=10313: Wed '
                                             'Oct 26 15:24:12 2022\n'
                                             '  read: IOPS=427k, BW=208MiB/s '
                                             '(218MB/s)(1458MiB/7001msec)\n'
                                             '    slat (nsec): min=1222, '
                                             'max=37791, avg=1814.16, '
                                             'stdev=388.86\n'
                                             '    clat (nsec): min=121, '
                                             'max=127819, avg=16622.12, '
                                             'stdev=3687.18\n'
                                             '     lat (usec): min=6, max=129, '
                                             'avg=18.44, stdev= 3.66\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[12608],  '
                                             '5.00th=[13248], 10.00th=[14144], '
                                             '20.00th=[15296],\n'
                                             '     | 30.00th=[15680], '
                                             '40.00th=[16064], '
                                             '50.00th=[16192], '
                                             '60.00th=[16512],\n'
                                             '     | 70.00th=[16768], '
                                             '80.00th=[17280], '
                                             '90.00th=[17536], '
                                             '95.00th=[18560],\n'
                                             '     | 99.00th=[36608], '
                                             '99.50th=[38656], '
                                             '99.90th=[48896], '
                                             '99.95th=[65280],\n'
                                             '     | 99.99th=[80384]\n'
                                             '   bw (  KiB/s): min=212347, '
                                             'max=214606, per=100.00%, '
                                             'avg=213309.14, stdev=669.71, '
                                             'samples=14\n'
                                             '   iops        : min=424694, '
                                             'max=429212, avg=426618.29, '
                                             'stdev=1339.43, samples=14\n'
                                             '  lat (nsec)   : 250=0.01%\n'
                                             '  lat (usec)   : 10=0.04%, '
                                             '20=96.30%, 50=3.57%, 100=0.10%, '
                                             '250=0.01%\n'
                                             '  cpu          : usr=21.26%, '
                                             'sys=75.81%, ctx=14684, majf=0, '
                                             'minf=1\n'
                                             '  IO depths    : 1=0.0%, 2=0.0%, '
                                             '4=0.0%, 8=100.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.1%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=2986215,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=8\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=208MiB/s (218MB/s), '
                                             '208MiB/s-208MiB/s '
                                             '(218MB/s-218MB/s), io=1458MiB '
                                             '(1529MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_315.output'),
                                   'state': {'begin': 1666783562.7305648,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_8_io_uring_cmd-xnvme_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8288524150848389,
                                             'end': 1666783563.5594172,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_315.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_315.state')},
                       'cmd_316': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_512_8_io_uring_cmd-bdev_xnvme_0.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_316.output'),
                                   'state': {'begin': 1666783565.6272268,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_8_io_uring_cmd-bdev_xnvme_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8053641319274902,
                                             'end': 1666783566.432591,
                                             'err': 1,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_316.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_316.state')},
                       'cmd_317': {'output': 'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_317.output'),
                                   'state': {'begin': 1666783566.4348829,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring_cmd-bdev_xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="8" --bs="512" '
                                                    '--direct="0" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="spdk_bdev" '
                                                    '--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" '
                                                    '--filename="Nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_8_io_uring_cmd-bdev_xnvme_0.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.552487850189209,
                                             'end': 1666783577.9873707,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_317.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_317.state')},
                       'cmd_318': {'output': 'io_uring_cmd-bdev_xnvme: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=spdk_bdev, iodepth=8\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666790667,\n'
                                             '  "timestamp_ms" : '
                                             '1666790667990,\n'
                                             '  "time" : "Wed Oct 26 15:24:27 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring_cmd-bdev_xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring_cmd-bdev_xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "8",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "0",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"spdk_bdev",\n'
                                             '        "spdk_conf" : '
                                             '"/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",\n'
                                             '        "filename" : "Nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '2123516928,\n'
                                             '        "io_kbytes" : 2073747,\n'
                                             '        "bw_bytes" : 303316230,\n'
                                             '        "bw" : 296207,\n'
                                             '        "iops" : 592413.512355,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 4147487,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 190,\n'
                                             '          "max" : 26219,\n'
                                             '          "mean" : 278.529163,\n'
                                             '          "stddev" : 75.839500,\n'
                                             '          "N" : 4147488\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 691,\n'
                                             '          "max" : 131106,\n'
                                             '          "mean" : '
                                             '12536.847786,\n'
                                             '          "stddev" : '
                                             '4087.236657,\n'
                                             '          "N" : 4147494,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 7008,\n'
                                             '            "5.000000" : 7904,\n'
                                             '            "10.000000" : 8640,\n'
                                             '            "20.000000" : 9664,\n'
                                             '            "30.000000" : '
                                             '10688,\n'
                                             '            "40.000000" : '
                                             '11456,\n'
                                             '            "50.000000" : '
                                             '12096,\n'
                                             '            "60.000000" : '
                                             '12608,\n'
                                             '            "70.000000" : '
                                             '13504,\n'
                                             '            "80.000000" : '
                                             '14656,\n'
                                             '            "90.000000" : '
                                             '15936,\n'
                                             '            "95.000000" : '
                                             '17280,\n'
                                             '            "99.000000" : '
                                             '29568,\n'
                                             '            "99.500000" : '
                                             '32128,\n'
                                             '            "99.900000" : '
                                             '43776,\n'
                                             '            "99.950000" : '
                                             '62720,\n'
                                             '            "99.990000" : 76288\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 5720,\n'
                                             '          "max" : 131396,\n'
                                             '          "mean" : '
                                             '12815.376954,\n'
                                             '          "stddev" : '
                                             '4084.439124,\n'
                                             '          "N" : 4147494\n'
                                             '        },\n'
                                             '        "bw_min" : 295554,\n'
                                             '        "bw_max" : 296921,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '296335.857143,\n'
                                             '        "bw_dev" : 340.016483,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 591108,\n'
                                             '        "iops_max" : 593843,\n'
                                             '        "iops_mean" : '
                                             '592671.857143,\n'
                                             '        "iops_stddev" : '
                                             '680.086969,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 99.985714,\n'
                                             '      "sys_cpu" : 0.000000,\n'
                                             '      "ctx" : 21,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.100000,\n'
                                             '        "2" : 0.100000,\n'
                                             '        "4" : 35.838135,\n'
                                             '        "8" : 64.137537,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 99.802206,\n'
                                             '        "8" : 0.197794,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.010000,\n'
                                             '        "1000" : 0.010000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.010000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 23.339482,\n'
                                             '        "20" : 73.205245,\n'
                                             '        "50" : 3.369510,\n'
                                             '        "100" : 0.084557,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 8,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring_cmd-bdev_xnvme: '
                                             '(groupid=0, jobs=1): err= 0: '
                                             'pid=10371: Wed Oct 26 15:24:27 '
                                             '2022\n'
                                             '  read: IOPS=592k, BW=289MiB/s '
                                             '(303MB/s)(2025MiB/7001msec)\n'
                                             '    slat (nsec): min=190, '
                                             'max=26219, avg=278.53, '
                                             'stdev=75.84\n'
                                             '    clat (nsec): min=691, '
                                             'max=131106, avg=12536.85, '
                                             'stdev=4087.24\n'
                                             '     lat (usec): min=5, max=131, '
                                             'avg=12.82, stdev= 4.08\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 7008],  '
                                             '5.00th=[ 7904], 10.00th=[ 8640], '
                                             '20.00th=[ 9664],\n'
                                             '     | 30.00th=[10688], '
                                             '40.00th=[11456], '
                                             '50.00th=[12096], '
                                             '60.00th=[12608],\n'
                                             '     | 70.00th=[13504], '
                                             '80.00th=[14656], '
                                             '90.00th=[15936], '
                                             '95.00th=[17280],\n'
                                             '     | 99.00th=[29568], '
                                             '99.50th=[32128], '
                                             '99.90th=[43776], '
                                             '99.95th=[62720],\n'
                                             '     | 99.99th=[76288]\n'
                                             '   bw (  KiB/s): min=295554, '
                                             'max=296921, per=100.00%, '
                                             'avg=296335.86, stdev=340.02, '
                                             'samples=14\n'
                                             '   iops        : min=591108, '
                                             'max=593843, avg=592671.86, '
                                             'stdev=680.09, samples=14\n'
                                             '  lat (nsec)   : 750=0.01%, '
                                             '1000=0.01%\n'
                                             '  lat (usec)   : 2=0.01%, '
                                             '4=0.01%, 10=23.34%, 20=73.21%, '
                                             '50=3.37%\n'
                                             '  lat (usec)   : 100=0.08%, '
                                             '250=0.01%\n'
                                             '  cpu          : usr=99.99%, '
                                             'sys=0.00%, ctx=21, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.1%, 2=0.1%, '
                                             '4=35.8%, 8=64.1%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=99.8%, 8=0.2%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=4147487,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=8\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=289MiB/s (303MB/s), '
                                             '289MiB/s-289MiB/s '
                                             '(303MB/s-303MB/s), io=2025MiB '
                                             '(2124MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_318.output'),
                                   'state': {'begin': 1666783577.9900692,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_8_io_uring_cmd-bdev_xnvme_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.9011662006378174,
                                             'end': 1666783578.8912354,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_318.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_318.state')},
                       'cmd_319': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_512_8_io_uring_cmd-bdev_xnvme_1.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_319.output'),
                                   'state': {'begin': 1666783581.059943,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_8_io_uring_cmd-bdev_xnvme_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8159592151641846,
                                             'end': 1666783581.8759022,
                                             'err': 1,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_319.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_319.state')},
                       'cmd_32': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_32.output'),
                                  'state': {'begin': 1666782189.9855325,
                                            'cmd': '/opt/fio/bin/fio '
                                                   '--name="io_uring-reference" '
                                                   '--rw="randread" '
                                                   '--size="12G" --iodepth="1" '
                                                   '--bs="512" --direct="1" '
                                                   '--thread="1" '
                                                   '--time_based="1" '
                                                   '--runtime="7" '
                                                   '--ramp_time="3" '
                                                   '--norandommap="1" '
                                                   '--allow_file_create="0" '
                                                   '--ioengine="io_uring" '
                                                   '--filename="/dev/nvme0n1" '
                                                   '--output-format="normal,json" '
                                                   '--output="/tmp/fio-output_512_1_io_uring-reference_1.txt" '
                                                   '--cpus_allowed="1"',
                                            'cwd': 'None',
                                            'elapsed': 11.055187463760376,
                                            'end': 1666782201.04072,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_32.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_32.state')},
                       'cmd_320': {'output': 'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_320.output'),
                                   'state': {'begin': 1666783581.87793,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring_cmd-bdev_xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="8" --bs="512" '
                                                    '--direct="0" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="spdk_bdev" '
                                                    '--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" '
                                                    '--filename="Nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_8_io_uring_cmd-bdev_xnvme_1.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.513900995254517,
                                             'end': 1666783593.391831,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_320.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_320.state')},
                       'cmd_321': {'output': 'io_uring_cmd-bdev_xnvme: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=spdk_bdev, iodepth=8\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666790683,\n'
                                             '  "timestamp_ms" : '
                                             '1666790683397,\n'
                                             '  "time" : "Wed Oct 26 15:24:43 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring_cmd-bdev_xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring_cmd-bdev_xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "8",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "0",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"spdk_bdev",\n'
                                             '        "spdk_conf" : '
                                             '"/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",\n'
                                             '        "filename" : "Nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '1576720896,\n'
                                             '        "io_kbytes" : 1539766,\n'
                                             '        "bw_bytes" : 225213668,\n'
                                             '        "bw" : 219935,\n'
                                             '        "iops" : 439869.447222,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 3079526,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 170,\n'
                                             '          "max" : 30026,\n'
                                             '          "mean" : 708.241025,\n'
                                             '          "stddev" : '
                                             '279.522443,\n'
                                             '          "N" : 3079527\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 711,\n'
                                             '          "max" : 200556,\n'
                                             '          "mean" : '
                                             '16166.595282,\n'
                                             '          "stddev" : '
                                             '5857.765495,\n'
                                             '          "N" : 3079533,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 7456,\n'
                                             '            "5.000000" : 8768,\n'
                                             '            "10.000000" : 9408,\n'
                                             '            "20.000000" : '
                                             '11072,\n'
                                             '            "30.000000" : '
                                             '12480,\n'
                                             '            "40.000000" : '
                                             '13632,\n'
                                             '            "50.000000" : '
                                             '15168,\n'
                                             '            "60.000000" : '
                                             '16768,\n'
                                             '            "70.000000" : '
                                             '18816,\n'
                                             '            "80.000000" : '
                                             '21376,\n'
                                             '            "90.000000" : '
                                             '23936,\n'
                                             '            "95.000000" : '
                                             '25728,\n'
                                             '            "99.000000" : '
                                             '33024,\n'
                                             '            "99.500000" : '
                                             '36096,\n'
                                             '            "99.900000" : '
                                             '47360,\n'
                                             '            "99.950000" : '
                                             '64256,\n'
                                             '            "99.990000" : 77312\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 5370,\n'
                                             '          "max" : 202119,\n'
                                             '          "mean" : '
                                             '16874.835481,\n'
                                             '          "stddev" : '
                                             '5927.145293,\n'
                                             '          "N" : 3079533\n'
                                             '        },\n'
                                             '        "bw_min" : 198922,\n'
                                             '        "bw_max" : 296648,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '220001.714286,\n'
                                             '        "bw_dev" : '
                                             '36653.514516,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 397844,\n'
                                             '        "iops_max" : 593296,\n'
                                             '        "iops_mean" : '
                                             '440003.428571,\n'
                                             '        "iops_stddev" : '
                                             '73307.214011,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 99.985714,\n'
                                             '      "sys_cpu" : 0.000000,\n'
                                             '      "ctx" : 21,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.100000,\n'
                                             '        "2" : 3.569933,\n'
                                             '        "4" : 45.009070,\n'
                                             '        "8" : 51.419309,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 97.162062,\n'
                                             '        "8" : 2.837938,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.010000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.010000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 13.485322,\n'
                                             '        "20" : 61.459426,\n'
                                             '        "50" : 24.963355,\n'
                                             '        "100" : 0.090858,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 8,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring_cmd-bdev_xnvme: '
                                             '(groupid=0, jobs=1): err= 0: '
                                             'pid=10431: Wed Oct 26 15:24:43 '
                                             '2022\n'
                                             '  read: IOPS=440k, BW=215MiB/s '
                                             '(225MB/s)(1504MiB/7001msec)\n'
                                             '    slat (nsec): min=170, '
                                             'max=30026, avg=708.24, '
                                             'stdev=279.52\n'
                                             '    clat (nsec): min=711, '
                                             'max=200556, avg=16166.60, '
                                             'stdev=5857.77\n'
                                             '     lat (usec): min=5, max=202, '
                                             'avg=16.87, stdev= 5.93\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 7456],  '
                                             '5.00th=[ 8768], 10.00th=[ 9408], '
                                             '20.00th=[11072],\n'
                                             '     | 30.00th=[12480], '
                                             '40.00th=[13632], '
                                             '50.00th=[15168], '
                                             '60.00th=[16768],\n'
                                             '     | 70.00th=[18816], '
                                             '80.00th=[21376], '
                                             '90.00th=[23936], '
                                             '95.00th=[25728],\n'
                                             '     | 99.00th=[33024], '
                                             '99.50th=[36096], '
                                             '99.90th=[47360], '
                                             '99.95th=[64256],\n'
                                             '     | 99.99th=[77312]\n'
                                             '   bw (  KiB/s): min=198922, '
                                             'max=296648, per=100.00%, '
                                             'avg=220001.71, stdev=36653.51, '
                                             'samples=14\n'
                                             '   iops        : min=397844, '
                                             'max=593296, avg=440003.43, '
                                             'stdev=73307.21, samples=14\n'
                                             '  lat (nsec)   : 750=0.01%\n'
                                             '  lat (usec)   : 2=0.01%, '
                                             '10=13.49%, 20=61.46%, 50=24.96%, '
                                             '100=0.09%\n'
                                             '  lat (usec)   : 250=0.01%\n'
                                             '  cpu          : usr=99.99%, '
                                             'sys=0.00%, ctx=21, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.1%, 2=3.6%, '
                                             '4=45.0%, 8=51.4%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=97.2%, 8=2.8%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=3079526,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=8\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=215MiB/s (225MB/s), '
                                             '215MiB/s-215MiB/s '
                                             '(225MB/s-225MB/s), io=1504MiB '
                                             '(1577MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_321.output'),
                                   'state': {'begin': 1666783593.3944402,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_8_io_uring_cmd-bdev_xnvme_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8707568645477295,
                                             'end': 1666783594.265197,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_321.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_321.state')},
                       'cmd_322': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_512_8_io_uring_cmd-bdev_xnvme_2.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_322.output'),
                                   'state': {'begin': 1666783596.4025614,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_8_io_uring_cmd-bdev_xnvme_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8284289836883545,
                                             'end': 1666783597.2309904,
                                             'err': 1,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_322.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_322.state')},
                       'cmd_323': {'output': 'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_323.output'),
                                   'state': {'begin': 1666783597.2333243,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring_cmd-bdev_xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="8" --bs="512" '
                                                    '--direct="0" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="spdk_bdev" '
                                                    '--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" '
                                                    '--filename="Nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_8_io_uring_cmd-bdev_xnvme_2.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.516216039657593,
                                             'end': 1666783608.7495403,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_323.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_323.state')},
                       'cmd_324': {'output': 'io_uring_cmd-bdev_xnvme: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=spdk_bdev, iodepth=8\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666790698,\n'
                                             '  "timestamp_ms" : '
                                             '1666790698756,\n'
                                             '  "time" : "Wed Oct 26 15:24:58 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring_cmd-bdev_xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring_cmd-bdev_xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "8",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "0",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"spdk_bdev",\n'
                                             '        "spdk_conf" : '
                                             '"/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",\n'
                                             '        "filename" : "Nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '1556664832,\n'
                                             '        "io_kbytes" : 1520180,\n'
                                             '        "bw_bytes" : 222348926,\n'
                                             '        "bw" : 217137,\n'
                                             '        "iops" : 434274.246536,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 3040354,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 631,\n'
                                             '          "max" : 14688,\n'
                                             '          "mean" : 866.610232,\n'
                                             '          "stddev" : '
                                             '144.688161,\n'
                                             '          "N" : 3040354\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 6462,\n'
                                             '          "max" : 132108,\n'
                                             '          "mean" : '
                                             '16360.181632,\n'
                                             '          "stddev" : '
                                             '5137.436929,\n'
                                             '          "N" : 3040360,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 8384,\n'
                                             '            "5.000000" : 9536,\n'
                                             '            "10.000000" : '
                                             '10304,\n'
                                             '            "20.000000" : '
                                             '12224,\n'
                                             '            "30.000000" : '
                                             '13376,\n'
                                             '            "40.000000" : '
                                             '14528,\n'
                                             '            "50.000000" : '
                                             '15680,\n'
                                             '            "60.000000" : '
                                             '17024,\n'
                                             '            "70.000000" : '
                                             '18816,\n'
                                             '            "80.000000" : '
                                             '20096,\n'
                                             '            "90.000000" : '
                                             '22144,\n'
                                             '            "95.000000" : '
                                             '24960,\n'
                                             '            "99.000000" : '
                                             '32640,\n'
                                             '            "99.500000" : '
                                             '36096,\n'
                                             '            "99.900000" : '
                                             '44800,\n'
                                             '            "99.950000" : '
                                             '61696,\n'
                                             '            "99.990000" : 76288\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 7364,\n'
                                             '          "max" : 132779,\n'
                                             '          "mean" : '
                                             '17226.791887,\n'
                                             '          "stddev" : '
                                             '5119.625326,\n'
                                             '          "N" : 3040360\n'
                                             '        },\n'
                                             '        "bw_min" : 216825,\n'
                                             '        "bw_max" : 217381,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '217176.142857,\n'
                                             '        "bw_dev" : 164.800045,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 433650,\n'
                                             '        "iops_max" : 434762,\n'
                                             '        "iops_mean" : '
                                             '434352.357143,\n'
                                             '        "iops_stddev" : '
                                             '329.733134,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 99.985714,\n'
                                             '      "sys_cpu" : 0.000000,\n'
                                             '      "ctx" : 16,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.100000,\n'
                                             '        "2" : 1.185553,\n'
                                             '        "4" : 45.805258,\n'
                                             '        "8" : 53.008959,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 98.136979,\n'
                                             '        "8" : 1.863021,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 8.138000,\n'
                                             '        "20" : 70.711042,\n'
                                             '        "50" : 21.071198,\n'
                                             '        "100" : 0.078215,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 8,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring_cmd-bdev_xnvme: '
                                             '(groupid=0, jobs=1): err= 0: '
                                             'pid=10490: Wed Oct 26 15:24:58 '
                                             '2022\n'
                                             '  read: IOPS=434k, BW=212MiB/s '
                                             '(222MB/s)(1485MiB/7001msec)\n'
                                             '    slat (nsec): min=631, '
                                             'max=14688, avg=866.61, '
                                             'stdev=144.69\n'
                                             '    clat (usec): min=6, max=132, '
                                             'avg=16.36, stdev= 5.14\n'
                                             '     lat (usec): min=7, max=132, '
                                             'avg=17.23, stdev= 5.12\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 8384],  '
                                             '5.00th=[ 9536], 10.00th=[10304], '
                                             '20.00th=[12224],\n'
                                             '     | 30.00th=[13376], '
                                             '40.00th=[14528], '
                                             '50.00th=[15680], '
                                             '60.00th=[17024],\n'
                                             '     | 70.00th=[18816], '
                                             '80.00th=[20096], '
                                             '90.00th=[22144], '
                                             '95.00th=[24960],\n'
                                             '     | 99.00th=[32640], '
                                             '99.50th=[36096], '
                                             '99.90th=[44800], '
                                             '99.95th=[61696],\n'
                                             '     | 99.99th=[76288]\n'
                                             '   bw (  KiB/s): min=216825, '
                                             'max=217381, per=100.00%, '
                                             'avg=217176.14, stdev=164.80, '
                                             'samples=14\n'
                                             '   iops        : min=433650, '
                                             'max=434762, avg=434352.36, '
                                             'stdev=329.73, samples=14\n'
                                             '  lat (usec)   : 10=8.14%, '
                                             '20=70.71%, 50=21.07%, 100=0.08%, '
                                             '250=0.01%\n'
                                             '  cpu          : usr=99.99%, '
                                             'sys=0.00%, ctx=16, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.1%, 2=1.2%, '
                                             '4=45.8%, 8=53.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=98.1%, 8=1.9%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=3040354,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=8\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=212MiB/s (222MB/s), '
                                             '212MiB/s-212MiB/s '
                                             '(222MB/s-222MB/s), io=1485MiB '
                                             '(1557MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_324.output'),
                                   'state': {'begin': 1666783608.752831,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_8_io_uring_cmd-bdev_xnvme_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8473482131958008,
                                             'end': 1666783609.6001792,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_324.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_324.state')},
                       'cmd_325': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_512_8_io_uring-reference_0.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_325.output'),
                                   'state': {'begin': 1666783610.610584,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_8_io_uring-reference_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8282756805419922,
                                             'end': 1666783611.4388597,
                                             'err': 1,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_325.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_325.state')},
                       'cmd_326': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_326.output'),
                                   'state': {'begin': 1666783611.4411962,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring-reference" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="8" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="io_uring" '
                                                    '--filename="/dev/nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_8_io_uring-reference_0.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.043455839157104,
                                             'end': 1666783622.484652,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_326.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_326.state')},
                       'cmd_327': {'output': 'io_uring-reference: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=io_uring, iodepth=8\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666790712,\n'
                                             '  "timestamp_ms" : '
                                             '1666790712555,\n'
                                             '  "time" : "Wed Oct 26 15:25:12 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring-reference",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring-reference",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "8",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"io_uring",\n'
                                             '        "filename" : '
                                             '"/dev/nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '1552151040,\n'
                                             '        "io_kbytes" : 1515772,\n'
                                             '        "bw_bytes" : 221704190,\n'
                                             '        "bw" : 216507,\n'
                                             '        "iops" : 433014.855021,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 3031537,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1332,\n'
                                             '          "max" : 31810,\n'
                                             '          "mean" : 1862.089115,\n'
                                             '          "stddev" : '
                                             '370.306038,\n'
                                             '          "N" : 3031538\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 2144,\n'
                                             '          "max" : 123952,\n'
                                             '          "mean" : '
                                             '16334.740680,\n'
                                             '          "stddev" : '
                                             '3609.143862,\n'
                                             '          "N" : 3031544,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 12480,\n'
                                             '            "5.000000" : 12992,\n'
                                             '            "10.000000" : '
                                             '13248,\n'
                                             '            "20.000000" : '
                                             '14784,\n'
                                             '            "30.000000" : '
                                             '15296,\n'
                                             '            "40.000000" : '
                                             '15552,\n'
                                             '            "50.000000" : '
                                             '15936,\n'
                                             '            "60.000000" : '
                                             '16192,\n'
                                             '            "70.000000" : '
                                             '16768,\n'
                                             '            "80.000000" : '
                                             '17280,\n'
                                             '            "90.000000" : '
                                             '17792,\n'
                                             '            "95.000000" : '
                                             '18816,\n'
                                             '            "99.000000" : '
                                             '32640,\n'
                                             '            "99.500000" : '
                                             '35072,\n'
                                             '            "99.900000" : '
                                             '51456,\n'
                                             '            "99.950000" : '
                                             '67072,\n'
                                             '            "99.990000" : 78336\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 7133,\n'
                                             '          "max" : 125345,\n'
                                             '          "mean" : '
                                             '18196.829907,\n'
                                             '          "stddev" : '
                                             '3573.540743,\n'
                                             '          "N" : 3031544\n'
                                             '        },\n'
                                             '        "bw_min" : 215528,\n'
                                             '        "bw_max" : 217965,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '216654.571429,\n'
                                             '        "bw_dev" : 805.018749,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 431056,\n'
                                             '        "iops_max" : 435931,\n'
                                             '        "iops_mean" : '
                                             '433309.285714,\n'
                                             '        "iops_stddev" : '
                                             '1610.180087,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 16.757143,\n'
                                             '      "sys_cpu" : 80.328571,\n'
                                             '      "ctx" : 15022,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 100.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 99.999967,\n'
                                             '        "8" : 0.100000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 0.056506,\n'
                                             '        "20" : 95.943741,\n'
                                             '        "50" : 3.889281,\n'
                                             '        "100" : 0.109647,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 8,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ],\n'
                                             '  "disk_util" : [\n'
                                             '    {\n'
                                             '      "name" : "nvme0n1",\n'
                                             '      "read_ios" : 4373344,\n'
                                             '      "write_ios" : 0,\n'
                                             '      "read_merges" : 0,\n'
                                             '      "write_merges" : 0,\n'
                                             '      "read_ticks" : 25161,\n'
                                             '      "write_ticks" : 0,\n'
                                             '      "in_queue" : 25161,\n'
                                             '      "util" : 99.108381\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring-reference: (groupid=0, '
                                             'jobs=1): err= 0: pid=10537: Wed '
                                             'Oct 26 15:25:12 2022\n'
                                             '  read: IOPS=433k, BW=211MiB/s '
                                             '(222MB/s)(1480MiB/7001msec)\n'
                                             '    slat (nsec): min=1332, '
                                             'max=31810, avg=1862.09, '
                                             'stdev=370.31\n'
                                             '    clat (usec): min=2, max=123, '
                                             'avg=16.33, stdev= 3.61\n'
                                             '     lat (usec): min=7, max=125, '
                                             'avg=18.20, stdev= 3.57\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[12480],  '
                                             '5.00th=[12992], 10.00th=[13248], '
                                             '20.00th=[14784],\n'
                                             '     | 30.00th=[15296], '
                                             '40.00th=[15552], '
                                             '50.00th=[15936], '
                                             '60.00th=[16192],\n'
                                             '     | 70.00th=[16768], '
                                             '80.00th=[17280], '
                                             '90.00th=[17792], '
                                             '95.00th=[18816],\n'
                                             '     | 99.00th=[32640], '
                                             '99.50th=[35072], '
                                             '99.90th=[51456], '
                                             '99.95th=[67072],\n'
                                             '     | 99.99th=[78336]\n'
                                             '   bw (  KiB/s): min=215528, '
                                             'max=217965, per=100.00%, '
                                             'avg=216654.57, stdev=805.02, '
                                             'samples=14\n'
                                             '   iops        : min=431056, '
                                             'max=435931, avg=433309.29, '
                                             'stdev=1610.18, samples=14\n'
                                             '  lat (usec)   : 4=0.01%, '
                                             '10=0.06%, 20=95.94%, 50=3.89%, '
                                             '100=0.11%\n'
                                             '  lat (usec)   : 250=0.01%\n'
                                             '  cpu          : usr=16.76%, '
                                             'sys=80.33%, ctx=15022, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, 2=0.0%, '
                                             '4=0.0%, 8=100.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.1%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=3031537,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=8\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=211MiB/s (222MB/s), '
                                             '211MiB/s-211MiB/s '
                                             '(222MB/s-222MB/s), io=1480MiB '
                                             '(1552MB), run=7001-7001msec\n'
                                             '\n'
                                             'Disk stats (read/write):\n'
                                             '  nvme0n1: ios=4373344/0, '
                                             'merge=0/0, ticks=25161/0, '
                                             'in_queue=25161, util=99.11%\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_327.output'),
                                   'state': {'begin': 1666783622.4870234,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_8_io_uring-reference_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.860776424407959,
                                             'end': 1666783623.3477998,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_327.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_327.state')},
                       'cmd_328': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_512_8_io_uring-reference_1.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_328.output'),
                                   'state': {'begin': 1666783624.3991337,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_8_io_uring-reference_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.850975513458252,
                                             'end': 1666783625.2501092,
                                             'err': 1,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_328.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_328.state')},
                       'cmd_329': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_329.output'),
                                   'state': {'begin': 1666783625.2517607,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring-reference" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="8" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="io_uring" '
                                                    '--filename="/dev/nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_8_io_uring-reference_1.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.097941875457764,
                                             'end': 1666783636.3497026,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_329.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_329.state')},
                       'cmd_33': {'output': 'io_uring-reference: (g=0): '
                                            'rw=randread, bs=(R) 512B-512B, '
                                            '(W) 512B-512B, (T) 512B-512B, '
                                            'ioengine=io_uring, iodepth=1\n'
                                            'fio-3.32-dirty\n'
                                            'Starting 1 thread\n'
                                            '{\n'
                                            '  "fio version" : '
                                            '"fio-3.32-dirty",\n'
                                            '  "timestamp" : 1666789291,\n'
                                            '  "timestamp_ms" : '
                                            '1666789291136,\n'
                                            '  "time" : "Wed Oct 26 15:01:31 '
                                            '2022",\n'
                                            '  "jobs" : [\n'
                                            '    {\n'
                                            '      "jobname" : '
                                            '"io_uring-reference",\n'
                                            '      "groupid" : 0,\n'
                                            '      "error" : 0,\n'
                                            '      "eta" : 0,\n'
                                            '      "elapsed" : 11,\n'
                                            '      "job options" : {\n'
                                            '        "name" : '
                                            '"io_uring-reference",\n'
                                            '        "rw" : "randread",\n'
                                            '        "size" : "12G",\n'
                                            '        "iodepth" : "1",\n'
                                            '        "bs" : "512",\n'
                                            '        "direct" : "1",\n'
                                            '        "thread" : "1",\n'
                                            '        "time_based" : "1",\n'
                                            '        "runtime" : "7",\n'
                                            '        "ramp_time" : "3",\n'
                                            '        "norandommap" : "1",\n'
                                            '        "allow_file_create" : '
                                            '"0",\n'
                                            '        "ioengine" : "io_uring",\n'
                                            '        "filename" : '
                                            '"/dev/nvme0n1",\n'
                                            '        "cpus_allowed" : "1"\n'
                                            '      },\n'
                                            '      "read" : {\n'
                                            '        "io_bytes" : 392546816,\n'
                                            '        "io_kbytes" : 383346,\n'
                                            '        "bw_bytes" : 56070106,\n'
                                            '        "bw" : 54755,\n'
                                            '        "iops" : 109511.784031,\n'
                                            '        "runtime" : 7001,\n'
                                            '        "total_ios" : 766692,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 1452,\n'
                                            '          "max" : 7204,\n'
                                            '          "mean" : 1551.774812,\n'
                                            '          "stddev" : 73.369744,\n'
                                            '          "N" : 766693\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 1042,\n'
                                            '          "max" : 78077,\n'
                                            '          "mean" : 7262.353467,\n'
                                            '          "stddev" : 979.853703,\n'
                                            '          "N" : 766692,\n'
                                            '          "percentile" : {\n'
                                            '            "1.000000" : 6816,\n'
                                            '            "5.000000" : 6944,\n'
                                            '            "10.000000" : 7008,\n'
                                            '            "20.000000" : 7072,\n'
                                            '            "30.000000" : 7072,\n'
                                            '            "40.000000" : 7136,\n'
                                            '            "50.000000" : 7136,\n'
                                            '            "60.000000" : 7200,\n'
                                            '            "70.000000" : 7264,\n'
                                            '            "80.000000" : 7264,\n'
                                            '            "90.000000" : 7328,\n'
                                            '            "95.000000" : 7456,\n'
                                            '            "99.000000" : 10432,\n'
                                            '            "99.500000" : 13632,\n'
                                            '            "99.900000" : 15680,\n'
                                            '            "99.950000" : 15808,\n'
                                            '            "99.990000" : 58112\n'
                                            '          }\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 6743,\n'
                                            '          "max" : 79590,\n'
                                            '          "mean" : 8814.128316,\n'
                                            '          "stddev" : 984.427864,\n'
                                            '          "N" : 766692\n'
                                            '        },\n'
                                            '        "bw_min" : 54595,\n'
                                            '        "bw_max" : 55137,\n'
                                            '        "bw_agg" : 100.000000,\n'
                                            '        "bw_mean" : '
                                            '54770.357143,\n'
                                            '        "bw_dev" : 141.769915,\n'
                                            '        "bw_samples" : 14,\n'
                                            '        "iops_min" : 109190,\n'
                                            '        "iops_max" : 110274,\n'
                                            '        "iops_mean" : '
                                            '109540.714286,\n'
                                            '        "iops_stddev" : '
                                            '283.539830,\n'
                                            '        "iops_samples" : 14\n'
                                            '      },\n'
                                            '      "write" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "trim" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "sync" : {\n'
                                            '        "total_ios" : 0,\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        }\n'
                                            '      },\n'
                                            '      "job_runtime" : 7000,\n'
                                            '      "usr_cpu" : 8.571429,\n'
                                            '      "sys_cpu" : 34.971429,\n'
                                            '      "ctx" : 766689,\n'
                                            '      "majf" : 0,\n'
                                            '      "minf" : 0,\n'
                                            '      "iodepth_level" : {\n'
                                            '        "1" : 100.000000,\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_submit" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_complete" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ns" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_us" : {\n'
                                            '        "2" : 0.010000,\n'
                                            '        "4" : 0.010000,\n'
                                            '        "10" : 98.723738,\n'
                                            '        "20" : 1.253959,\n'
                                            '        "50" : 0.010000,\n'
                                            '        "100" : 0.014217,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ms" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000,\n'
                                            '        "2000" : 0.000000,\n'
                                            '        ">=2000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_depth" : 1,\n'
                                            '      "latency_target" : 0,\n'
                                            '      "latency_percentile" : '
                                            '100.000000,\n'
                                            '      "latency_window" : 0\n'
                                            '    }\n'
                                            '  ],\n'
                                            '  "disk_util" : [\n'
                                            '    {\n'
                                            '      "name" : "nvme0n1",\n'
                                            '      "read_ios" : 1098692,\n'
                                            '      "write_ios" : 0,\n'
                                            '      "read_merges" : 0,\n'
                                            '      "write_merges" : 0,\n'
                                            '      "read_ticks" : 5507,\n'
                                            '      "write_ticks" : 0,\n'
                                            '      "in_queue" : 5507,\n'
                                            '      "util" : 99.069123\n'
                                            '    }\n'
                                            '  ]\n'
                                            '}\n'
                                            '\n'
                                            'io_uring-reference: (groupid=0, '
                                            'jobs=1): err= 0: pid=5437: Wed '
                                            'Oct 26 15:01:31 2022\n'
                                            '  read: IOPS=110k, BW=53.5MiB/s '
                                            '(56.1MB/s)(374MiB/7001msec)\n'
                                            '    slat (nsec): min=1452, '
                                            'max=7204, avg=1551.77, '
                                            'stdev=73.37\n'
                                            '    clat (nsec): min=1042, '
                                            'max=78077, avg=7262.35, '
                                            'stdev=979.85\n'
                                            '     lat (nsec): min=6743, '
                                            'max=79590, avg=8814.13, '
                                            'stdev=984.43\n'
                                            '    clat percentiles (nsec):\n'
                                            '     |  1.00th=[ 6816],  5.00th=[ '
                                            '6944], 10.00th=[ 7008], 20.00th=[ '
                                            '7072],\n'
                                            '     | 30.00th=[ 7072], 40.00th=[ '
                                            '7136], 50.00th=[ 7136], 60.00th=[ '
                                            '7200],\n'
                                            '     | 70.00th=[ 7264], 80.00th=[ '
                                            '7264], 90.00th=[ 7328], 95.00th=[ '
                                            '7456],\n'
                                            '     | 99.00th=[10432], '
                                            '99.50th=[13632], 99.90th=[15680], '
                                            '99.95th=[15808],\n'
                                            '     | 99.99th=[58112]\n'
                                            '   bw (  KiB/s): min=54595, '
                                            'max=55137, per=100.00%, '
                                            'avg=54770.36, stdev=141.77, '
                                            'samples=14\n'
                                            '   iops        : min=109190, '
                                            'max=110274, avg=109540.71, '
                                            'stdev=283.54, samples=14\n'
                                            '  lat (usec)   : 2=0.01%, '
                                            '4=0.01%, 10=98.72%, 20=1.25%, '
                                            '50=0.01%\n'
                                            '  lat (usec)   : 100=0.01%\n'
                                            '  cpu          : usr=8.57%, '
                                            'sys=34.97%, ctx=766689, majf=0, '
                                            'minf=0\n'
                                            '  IO depths    : 1=100.0%, '
                                            '2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, >=64=0.0%\n'
                                            '     submit    : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     complete  : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     issued rwts: '
                                            'total=766692,0,0,0 short=0,0,0,0 '
                                            'dropped=0,0,0,0\n'
                                            '     latency   : target=0, '
                                            'window=0, percentile=100.00%, '
                                            'depth=1\n'
                                            '\n'
                                            'Run status group 0 (all jobs):\n'
                                            '   READ: bw=53.5MiB/s (56.1MB/s), '
                                            '53.5MiB/s-53.5MiB/s '
                                            '(56.1MB/s-56.1MB/s), io=374MiB '
                                            '(393MB), run=7001-7001msec\n'
                                            '\n'
                                            'Disk stats (read/write):\n'
                                            '  nvme0n1: ios=1098692/0, '
                                            'merge=0/0, ticks=5507/0, '
                                            'in_queue=5507, util=99.07%\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_33.output'),
                                  'state': {'begin': 1666782201.0473912,
                                            'cmd': 'cat '
                                                   '/tmp/fio-output_512_1_io_uring-reference_1.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.8552923202514648,
                                            'end': 1666782201.9026835,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_33.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_33.state')},
                       'cmd_330': {'output': 'io_uring-reference: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=io_uring, iodepth=8\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666790726,\n'
                                             '  "timestamp_ms" : '
                                             '1666790726420,\n'
                                             '  "time" : "Wed Oct 26 15:25:26 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring-reference",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring-reference",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "8",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"io_uring",\n'
                                             '        "filename" : '
                                             '"/dev/nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '1511606784,\n'
                                             '        "io_kbytes" : 1476178,\n'
                                             '        "bw_bytes" : 215912981,\n'
                                             '        "bw" : 210852,\n'
                                             '        "iops" : 421704.042280,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 2952350,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1333,\n'
                                             '          "max" : 30808,\n'
                                             '          "mean" : 1878.538164,\n'
                                             '          "stddev" : '
                                             '370.577820,\n'
                                             '          "N" : 2952350\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 3927,\n'
                                             '          "max" : 150492,\n'
                                             '          "mean" : '
                                             '16780.232220,\n'
                                             '          "stddev" : '
                                             '3545.257165,\n'
                                             '          "N" : 2952357,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 12736,\n'
                                             '            "5.000000" : 13248,\n'
                                             '            "10.000000" : '
                                             '13760,\n'
                                             '            "20.000000" : '
                                             '15168,\n'
                                             '            "30.000000" : '
                                             '15808,\n'
                                             '            "40.000000" : '
                                             '16192,\n'
                                             '            "50.000000" : '
                                             '16512,\n'
                                             '            "60.000000" : '
                                             '16768,\n'
                                             '            "70.000000" : '
                                             '17024,\n'
                                             '            "80.000000" : '
                                             '17536,\n'
                                             '            "90.000000" : '
                                             '18048,\n'
                                             '            "95.000000" : '
                                             '18816,\n'
                                             '            "99.000000" : '
                                             '33024,\n'
                                             '            "99.500000" : '
                                             '35072,\n'
                                             '            "99.900000" : '
                                             '50944,\n'
                                             '            "99.950000" : '
                                             '67072,\n'
                                             '            "99.990000" : 78336\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 7194,\n'
                                             '          "max" : 151965,\n'
                                             '          "mean" : '
                                             '18658.769599,\n'
                                             '          "stddev" : '
                                             '3523.940859,\n'
                                             '          "N" : 2952357\n'
                                             '        },\n'
                                             '        "bw_min" : 208714,\n'
                                             '        "bw_max" : 212848,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '210911.285714,\n'
                                             '        "bw_dev" : 1029.007022,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 417428,\n'
                                             '        "iops_max" : 425697,\n'
                                             '        "iops_mean" : '
                                             '421822.500000,\n'
                                             '        "iops_stddev" : '
                                             '2058.121277,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 15.842857,\n'
                                             '      "sys_cpu" : 81.442857,\n'
                                             '      "ctx" : 15090,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 100.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 99.999966,\n'
                                             '        "8" : 0.100000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 0.087591,\n'
                                             '        "20" : 95.854726,\n'
                                             '        "50" : 3.952004,\n'
                                             '        "100" : 0.104730,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 8,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ],\n'
                                             '  "disk_util" : [\n'
                                             '    {\n'
                                             '      "name" : "nvme0n1",\n'
                                             '      "read_ios" : 4279306,\n'
                                             '      "write_ios" : 0,\n'
                                             '      "read_merges" : 0,\n'
                                             '      "write_merges" : 0,\n'
                                             '      "read_ticks" : 24601,\n'
                                             '      "write_ticks" : 0,\n'
                                             '      "in_queue" : 24601,\n'
                                             '      "util" : 99.069123\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring-reference: (groupid=0, '
                                             'jobs=1): err= 0: pid=10583: Wed '
                                             'Oct 26 15:25:26 2022\n'
                                             '  read: IOPS=422k, BW=206MiB/s '
                                             '(216MB/s)(1442MiB/7001msec)\n'
                                             '    slat (nsec): min=1333, '
                                             'max=30808, avg=1878.54, '
                                             'stdev=370.58\n'
                                             '    clat (usec): min=3, max=150, '
                                             'avg=16.78, stdev= 3.55\n'
                                             '     lat (usec): min=7, max=151, '
                                             'avg=18.66, stdev= 3.52\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[12736],  '
                                             '5.00th=[13248], 10.00th=[13760], '
                                             '20.00th=[15168],\n'
                                             '     | 30.00th=[15808], '
                                             '40.00th=[16192], '
                                             '50.00th=[16512], '
                                             '60.00th=[16768],\n'
                                             '     | 70.00th=[17024], '
                                             '80.00th=[17536], '
                                             '90.00th=[18048], '
                                             '95.00th=[18816],\n'
                                             '     | 99.00th=[33024], '
                                             '99.50th=[35072], '
                                             '99.90th=[50944], '
                                             '99.95th=[67072],\n'
                                             '     | 99.99th=[78336]\n'
                                             '   bw (  KiB/s): min=208714, '
                                             'max=212848, per=100.00%, '
                                             'avg=210911.29, stdev=1029.01, '
                                             'samples=14\n'
                                             '   iops        : min=417428, '
                                             'max=425697, avg=421822.50, '
                                             'stdev=2058.12, samples=14\n'
                                             '  lat (usec)   : 4=0.01%, '
                                             '10=0.09%, 20=95.85%, 50=3.95%, '
                                             '100=0.10%\n'
                                             '  lat (usec)   : 250=0.01%\n'
                                             '  cpu          : usr=15.84%, '
                                             'sys=81.44%, ctx=15090, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, 2=0.0%, '
                                             '4=0.0%, 8=100.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.1%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=2952350,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=8\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=206MiB/s (216MB/s), '
                                             '206MiB/s-206MiB/s '
                                             '(216MB/s-216MB/s), io=1442MiB '
                                             '(1512MB), run=7001-7001msec\n'
                                             '\n'
                                             'Disk stats (read/write):\n'
                                             '  nvme0n1: ios=4279306/0, '
                                             'merge=0/0, ticks=24601/0, '
                                             'in_queue=24601, util=99.07%\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_330.output'),
                                   'state': {'begin': 1666783636.3520877,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_8_io_uring-reference_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8014988899230957,
                                             'end': 1666783637.1535866,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_330.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_330.state')},
                       'cmd_331': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_512_8_io_uring-reference_2.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_331.output'),
                                   'state': {'begin': 1666783638.214614,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_8_io_uring-reference_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8205971717834473,
                                             'end': 1666783639.035211,
                                             'err': 1,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_331.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_331.state')},
                       'cmd_332': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_332.output'),
                                   'state': {'begin': 1666783639.0374577,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring-reference" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="8" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="io_uring" '
                                                    '--filename="/dev/nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_8_io_uring-reference_2.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.052669763565063,
                                             'end': 1666783650.0901275,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_332.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_332.state')},
                       'cmd_333': {'output': 'io_uring-reference: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=io_uring, iodepth=8\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666790740,\n'
                                             '  "timestamp_ms" : '
                                             '1666790740160,\n'
                                             '  "time" : "Wed Oct 26 15:25:40 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring-reference",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring-reference",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "8",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"io_uring",\n'
                                             '        "filename" : '
                                             '"/dev/nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '1562410496,\n'
                                             '        "io_kbytes" : 1525791,\n'
                                             '        "bw_bytes" : 223169618,\n'
                                             '        "bw" : 217939,\n'
                                             '        "iops" : 435877.017569,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 3051575,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1322,\n'
                                             '          "max" : 37350,\n'
                                             '          "mean" : 1848.436398,\n'
                                             '          "stddev" : '
                                             '360.896377,\n'
                                             '          "N" : 3051576\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 5079,\n'
                                             '          "max" : 131407,\n'
                                             '          "mean" : '
                                             '16231.104474,\n'
                                             '          "stddev" : '
                                             '3574.667520,\n'
                                             '          "N" : 3051582,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 12480,\n'
                                             '            "5.000000" : 12992,\n'
                                             '            "10.000000" : '
                                             '13248,\n'
                                             '            "20.000000" : '
                                             '14784,\n'
                                             '            "30.000000" : '
                                             '15296,\n'
                                             '            "40.000000" : '
                                             '15424,\n'
                                             '            "50.000000" : '
                                             '15808,\n'
                                             '            "60.000000" : '
                                             '16064,\n'
                                             '            "70.000000" : '
                                             '16512,\n'
                                             '            "80.000000" : '
                                             '17280,\n'
                                             '            "90.000000" : '
                                             '17792,\n'
                                             '            "95.000000" : '
                                             '18560,\n'
                                             '            "99.000000" : '
                                             '32640,\n'
                                             '            "99.500000" : '
                                             '34560,\n'
                                             '            "99.900000" : '
                                             '49408,\n'
                                             '            "99.950000" : '
                                             '65280,\n'
                                             '            "99.990000" : 77312\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 6452,\n'
                                             '          "max" : 132769,\n'
                                             '          "mean" : '
                                             '18079.540984,\n'
                                             '          "stddev" : '
                                             '3537.875972,\n'
                                             '          "N" : 3051582\n'
                                             '        },\n'
                                             '        "bw_min" : 217144,\n'
                                             '        "bw_max" : 219141,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '217999.642857,\n'
                                             '        "bw_dev" : 593.977934,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 434288,\n'
                                             '        "iops_max" : 438282,\n'
                                             '        "iops_mean" : '
                                             '435999.357143,\n'
                                             '        "iops_stddev" : '
                                             '1188.013701,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 16.628571,\n'
                                             '      "sys_cpu" : 80.442857,\n'
                                             '      "ctx" : 14900,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 100.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 99.999967,\n'
                                             '        "8" : 0.100000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.058167,\n'
                                             '        "20" : 96.003277,\n'
                                             '        "50" : 3.840115,\n'
                                             '        "100" : 0.098146,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 8,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ],\n'
                                             '  "disk_util" : [\n'
                                             '    {\n'
                                             '      "name" : "nvme0n1",\n'
                                             '      "read_ios" : 4399333,\n'
                                             '      "write_ios" : 0,\n'
                                             '      "read_merges" : 0,\n'
                                             '      "write_merges" : 0,\n'
                                             '      "read_ticks" : 25278,\n'
                                             '      "write_ticks" : 0,\n'
                                             '      "in_queue" : 25278,\n'
                                             '      "util" : 99.069123\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring-reference: (groupid=0, '
                                             'jobs=1): err= 0: pid=10630: Wed '
                                             'Oct 26 15:25:40 2022\n'
                                             '  read: IOPS=436k, BW=213MiB/s '
                                             '(223MB/s)(1490MiB/7001msec)\n'
                                             '    slat (nsec): min=1322, '
                                             'max=37350, avg=1848.44, '
                                             'stdev=360.90\n'
                                             '    clat (usec): min=5, max=131, '
                                             'avg=16.23, stdev= 3.57\n'
                                             '     lat (usec): min=6, max=132, '
                                             'avg=18.08, stdev= 3.54\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[12480],  '
                                             '5.00th=[12992], 10.00th=[13248], '
                                             '20.00th=[14784],\n'
                                             '     | 30.00th=[15296], '
                                             '40.00th=[15424], '
                                             '50.00th=[15808], '
                                             '60.00th=[16064],\n'
                                             '     | 70.00th=[16512], '
                                             '80.00th=[17280], '
                                             '90.00th=[17792], '
                                             '95.00th=[18560],\n'
                                             '     | 99.00th=[32640], '
                                             '99.50th=[34560], '
                                             '99.90th=[49408], '
                                             '99.95th=[65280],\n'
                                             '     | 99.99th=[77312]\n'
                                             '   bw (  KiB/s): min=217144, '
                                             'max=219141, per=100.00%, '
                                             'avg=217999.64, stdev=593.98, '
                                             'samples=14\n'
                                             '   iops        : min=434288, '
                                             'max=438282, avg=435999.36, '
                                             'stdev=1188.01, samples=14\n'
                                             '  lat (usec)   : 10=0.06%, '
                                             '20=96.00%, 50=3.84%, 100=0.10%, '
                                             '250=0.01%\n'
                                             '  cpu          : usr=16.63%, '
                                             'sys=80.44%, ctx=14900, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, 2=0.0%, '
                                             '4=0.0%, 8=100.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.1%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=3051575,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=8\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=213MiB/s (223MB/s), '
                                             '213MiB/s-213MiB/s '
                                             '(223MB/s-223MB/s), io=1490MiB '
                                             '(1562MB), run=7001-7001msec\n'
                                             '\n'
                                             'Disk stats (read/write):\n'
                                             '  nvme0n1: ios=4399333/0, '
                                             'merge=0/0, ticks=25278/0, '
                                             'in_queue=25278, util=99.07%\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_333.output'),
                                   'state': {'begin': 1666783650.092708,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_8_io_uring-reference_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.842928409576416,
                                             'end': 1666783650.9356365,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_333.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_333.state')},
                       'cmd_334': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_512_8_io_uring-bdev_uring_0.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_334.output'),
                                   'state': {'begin': 1666783653.055604,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_8_io_uring-bdev_uring_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8298826217651367,
                                             'end': 1666783653.8854866,
                                             'err': 1,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_334.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_334.state')},
                       'cmd_335': {'output': 'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_335.output'),
                                   'state': {'begin': 1666783653.8873825,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring-bdev_uring" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="8" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="spdk_bdev" '
                                                    '--spdk_conf="/tmp/spdk_bdev_uring_foo.conf" '
                                                    '--filename="Nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_8_io_uring-bdev_uring_0.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.41184949874878,
                                             'end': 1666783665.299232,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_335.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_335.state')},
                       'cmd_336': {'output': 'io_uring-bdev_uring: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=spdk_bdev, iodepth=8\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666790755,\n'
                                             '  "timestamp_ms" : '
                                             '1666790755304,\n'
                                             '  "time" : "Wed Oct 26 15:25:55 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring-bdev_uring",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring-bdev_uring",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "8",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"spdk_bdev",\n'
                                             '        "spdk_conf" : '
                                             '"/tmp/spdk_bdev_uring_foo.conf",\n'
                                             '        "filename" : "Nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '1789819392,\n'
                                             '        "io_kbytes" : 1747870,\n'
                                             '        "bw_bytes" : 255651962,\n'
                                             '        "bw" : 249660,\n'
                                             '        "iops" : 499319.240109,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 3495734,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 110,\n'
                                             '          "max" : 23364,\n'
                                             '          "mean" : 152.689076,\n'
                                             '          "stddev" : '
                                             '114.264283,\n'
                                             '          "N" : 3495734\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 6803,\n'
                                             '          "max" : 131567,\n'
                                             '          "mean" : '
                                             '15002.125793,\n'
                                             '          "stddev" : '
                                             '4088.111547,\n'
                                             '          "N" : 3495738,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 9152,\n'
                                             '            "5.000000" : 12608,\n'
                                             '            "10.000000" : '
                                             '12864,\n'
                                             '            "20.000000" : '
                                             '13248,\n'
                                             '            "30.000000" : '
                                             '13504,\n'
                                             '            "40.000000" : '
                                             '13760,\n'
                                             '            "50.000000" : '
                                             '14016,\n'
                                             '            "60.000000" : '
                                             '14272,\n'
                                             '            "70.000000" : '
                                             '14656,\n'
                                             '            "80.000000" : '
                                             '15296,\n'
                                             '            "90.000000" : '
                                             '18304,\n'
                                             '            "95.000000" : '
                                             '21632,\n'
                                             '            "99.000000" : '
                                             '33024,\n'
                                             '            "99.500000" : '
                                             '34560,\n'
                                             '            "99.900000" : '
                                             '51968,\n'
                                             '            "99.950000" : '
                                             '67072,\n'
                                             '            "99.990000" : 78336\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 6973,\n'
                                             '          "max" : 131697,\n'
                                             '          "mean" : '
                                             '15154.814849,\n'
                                             '          "stddev" : '
                                             '4085.047977,\n'
                                             '          "N" : 3495738\n'
                                             '        },\n'
                                             '        "bw_min" : 247874,\n'
                                             '        "bw_max" : 252834,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '249794.071429,\n'
                                             '        "bw_dev" : 1473.101410,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 495748,\n'
                                             '        "iops_max" : 505668,\n'
                                             '        "iops_mean" : '
                                             '499588.285714,\n'
                                             '        "iops_stddev" : '
                                             '2946.178762,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 30.042857,\n'
                                             '      "sys_cpu" : 69.928571,\n'
                                             '      "ctx" : 8,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.100000,\n'
                                             '        "2" : 0.100000,\n'
                                             '        "4" : 72.377446,\n'
                                             '        "8" : 27.523175,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 98.713086,\n'
                                             '        "8" : 1.286914,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 1.834121,\n'
                                             '        "20" : 91.377834,\n'
                                             '        "50" : 6.677682,\n'
                                             '        "100" : 0.109390,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 8,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring-bdev_uring: (groupid=0, '
                                             'jobs=1): err= 0: pid=10686: Wed '
                                             'Oct 26 15:25:55 2022\n'
                                             '  read: IOPS=499k, BW=244MiB/s '
                                             '(256MB/s)(1707MiB/7001msec)\n'
                                             '    slat (nsec): min=110, '
                                             'max=23364, avg=152.69, '
                                             'stdev=114.26\n'
                                             '    clat (usec): min=6, max=131, '
                                             'avg=15.00, stdev= 4.09\n'
                                             '     lat (usec): min=6, max=131, '
                                             'avg=15.15, stdev= 4.09\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 9152],  '
                                             '5.00th=[12608], 10.00th=[12864], '
                                             '20.00th=[13248],\n'
                                             '     | 30.00th=[13504], '
                                             '40.00th=[13760], '
                                             '50.00th=[14016], '
                                             '60.00th=[14272],\n'
                                             '     | 70.00th=[14656], '
                                             '80.00th=[15296], '
                                             '90.00th=[18304], '
                                             '95.00th=[21632],\n'
                                             '     | 99.00th=[33024], '
                                             '99.50th=[34560], '
                                             '99.90th=[51968], '
                                             '99.95th=[67072],\n'
                                             '     | 99.99th=[78336]\n'
                                             '   bw (  KiB/s): min=247874, '
                                             'max=252834, per=100.00%, '
                                             'avg=249794.07, stdev=1473.10, '
                                             'samples=14\n'
                                             '   iops        : min=495748, '
                                             'max=505668, avg=499588.29, '
                                             'stdev=2946.18, samples=14\n'
                                             '  lat (usec)   : 10=1.83%, '
                                             '20=91.38%, 50=6.68%, 100=0.11%, '
                                             '250=0.01%\n'
                                             '  cpu          : usr=30.04%, '
                                             'sys=69.93%, ctx=8, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.1%, 2=0.1%, '
                                             '4=72.4%, 8=27.5%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=98.7%, 8=1.3%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=3495734,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=8\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=244MiB/s (256MB/s), '
                                             '244MiB/s-244MiB/s '
                                             '(256MB/s-256MB/s), io=1707MiB '
                                             '(1790MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_336.output'),
                                   'state': {'begin': 1666783665.3017976,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_8_io_uring-bdev_uring_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.79756760597229,
                                             'end': 1666783666.0993652,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_336.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_336.state')},
                       'cmd_337': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_512_8_io_uring-bdev_uring_1.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_337.output'),
                                   'state': {'begin': 1666783668.1847413,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_8_io_uring-bdev_uring_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8187205791473389,
                                             'end': 1666783669.0034618,
                                             'err': 1,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_337.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_337.state')},
                       'cmd_338': {'output': 'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_338.output'),
                                   'state': {'begin': 1666783669.0102723,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring-bdev_uring" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="8" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="spdk_bdev" '
                                                    '--spdk_conf="/tmp/spdk_bdev_uring_foo.conf" '
                                                    '--filename="Nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_8_io_uring-bdev_uring_1.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.410768747329712,
                                             'end': 1666783680.421041,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_338.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_338.state')},
                       'cmd_339': {'output': 'io_uring-bdev_uring: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=spdk_bdev, iodepth=8\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666790770,\n'
                                             '  "timestamp_ms" : '
                                             '1666790770427,\n'
                                             '  "time" : "Wed Oct 26 15:26:10 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring-bdev_uring",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring-bdev_uring",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "8",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"spdk_bdev",\n'
                                             '        "spdk_conf" : '
                                             '"/tmp/spdk_bdev_uring_foo.conf",\n'
                                             '        "filename" : "Nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '1791554560,\n'
                                             '        "io_kbytes" : 1749565,\n'
                                             '        "bw_bytes" : 255899808,\n'
                                             '        "bw" : 249902,\n'
                                             '        "iops" : 499803.313812,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 3499123,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 110,\n'
                                             '          "max" : 24446,\n'
                                             '          "mean" : 156.179399,\n'
                                             '          "stddev" : '
                                             '115.772836,\n'
                                             '          "N" : 3499123\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 6753,\n'
                                             '          "max" : 143268,\n'
                                             '          "mean" : '
                                             '14972.242013,\n'
                                             '          "stddev" : '
                                             '4066.042433,\n'
                                             '          "N" : 3499127,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 9152,\n'
                                             '            "5.000000" : 12480,\n'
                                             '            "10.000000" : '
                                             '12864,\n'
                                             '            "20.000000" : '
                                             '13248,\n'
                                             '            "30.000000" : '
                                             '13504,\n'
                                             '            "40.000000" : '
                                             '13760,\n'
                                             '            "50.000000" : '
                                             '14016,\n'
                                             '            "60.000000" : '
                                             '14272,\n'
                                             '            "70.000000" : '
                                             '14656,\n'
                                             '            "80.000000" : '
                                             '15168,\n'
                                             '            "90.000000" : '
                                             '18304,\n'
                                             '            "95.000000" : '
                                             '21632,\n'
                                             '            "99.000000" : '
                                             '33024,\n'
                                             '            "99.500000" : '
                                             '34560,\n'
                                             '            "99.900000" : '
                                             '50944,\n'
                                             '            "99.950000" : '
                                             '67072,\n'
                                             '            "99.990000" : 78336\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 6923,\n'
                                             '          "max" : 143469,\n'
                                             '          "mean" : '
                                             '15128.421388,\n'
                                             '          "stddev" : '
                                             '4063.050872,\n'
                                             '          "N" : 3499127\n'
                                             '        },\n'
                                             '        "bw_min" : 247329,\n'
                                             '        "bw_max" : 253134,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '250039.500000,\n'
                                             '        "bw_dev" : 1803.163577,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 494658,\n'
                                             '        "iops_max" : 506268,\n'
                                             '        "iops_mean" : '
                                             '500079.142857,\n'
                                             '        "iops_stddev" : '
                                             '3606.253582,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 29.642857,\n'
                                             '      "sys_cpu" : 70.328571,\n'
                                             '      "ctx" : 9,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.100000,\n'
                                             '        "2" : 0.117515,\n'
                                             '        "4" : 72.370934,\n'
                                             '        "8" : 27.508121,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 98.752327,\n'
                                             '        "8" : 1.247673,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 1.845948,\n'
                                             '        "20" : 91.447028,\n'
                                             '        "50" : 6.605484,\n'
                                             '        "100" : 0.100740,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 8,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring-bdev_uring: (groupid=0, '
                                             'jobs=1): err= 0: pid=10743: Wed '
                                             'Oct 26 15:26:10 2022\n'
                                             '  read: IOPS=500k, BW=244MiB/s '
                                             '(256MB/s)(1709MiB/7001msec)\n'
                                             '    slat (nsec): min=110, '
                                             'max=24446, avg=156.18, '
                                             'stdev=115.77\n'
                                             '    clat (usec): min=6, max=143, '
                                             'avg=14.97, stdev= 4.07\n'
                                             '     lat (usec): min=6, max=143, '
                                             'avg=15.13, stdev= 4.06\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 9152],  '
                                             '5.00th=[12480], 10.00th=[12864], '
                                             '20.00th=[13248],\n'
                                             '     | 30.00th=[13504], '
                                             '40.00th=[13760], '
                                             '50.00th=[14016], '
                                             '60.00th=[14272],\n'
                                             '     | 70.00th=[14656], '
                                             '80.00th=[15168], '
                                             '90.00th=[18304], '
                                             '95.00th=[21632],\n'
                                             '     | 99.00th=[33024], '
                                             '99.50th=[34560], '
                                             '99.90th=[50944], '
                                             '99.95th=[67072],\n'
                                             '     | 99.99th=[78336]\n'
                                             '   bw (  KiB/s): min=247329, '
                                             'max=253134, per=100.00%, '
                                             'avg=250039.50, stdev=1803.16, '
                                             'samples=14\n'
                                             '   iops        : min=494658, '
                                             'max=506268, avg=500079.14, '
                                             'stdev=3606.25, samples=14\n'
                                             '  lat (usec)   : 10=1.85%, '
                                             '20=91.45%, 50=6.61%, 100=0.10%, '
                                             '250=0.01%\n'
                                             '  cpu          : usr=29.64%, '
                                             'sys=70.33%, ctx=9, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.1%, 2=0.1%, '
                                             '4=72.4%, 8=27.5%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=98.8%, 8=1.2%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=3499123,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=8\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=244MiB/s (256MB/s), '
                                             '244MiB/s-244MiB/s '
                                             '(256MB/s-256MB/s), io=1709MiB '
                                             '(1792MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_339.output'),
                                   'state': {'begin': 1666783680.4237885,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_8_io_uring-bdev_uring_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 1.214738368988037,
                                             'end': 1666783681.638527,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_339.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_339.state')},
                       'cmd_34': {'output': 'rm: cannot remove '
                                            "'/tmp/fio-output_512_1_io_uring-reference_2.txt': "
                                            'No such file or directory\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_34.output'),
                                  'state': {'begin': 1666782202.960092,
                                            'cmd': 'rm '
                                                   '/tmp/fio-output_512_1_io_uring-reference_2.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.7774829864501953,
                                            'end': 1666782203.737575,
                                            'err': 1,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_34.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_34.state')},
                       'cmd_340': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_512_8_io_uring-bdev_uring_2.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_340.output'),
                                   'state': {'begin': 1666783685.4833064,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_8_io_uring-bdev_uring_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 1.1595447063446045,
                                             'end': 1666783686.642851,
                                             'err': 1,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_340.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_340.state')},
                       'cmd_341': {'output': 'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_341.output'),
                                   'state': {'begin': 1666783686.650046,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring-bdev_uring" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="8" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="spdk_bdev" '
                                                    '--spdk_conf="/tmp/spdk_bdev_uring_foo.conf" '
                                                    '--filename="Nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_8_io_uring-bdev_uring_2.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.890761375427246,
                                             'end': 1666783698.5408075,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_341.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_341.state')},
                       'cmd_342': {'output': 'io_uring-bdev_uring: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=spdk_bdev, iodepth=8\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666790788,\n'
                                             '  "timestamp_ms" : '
                                             '1666790788540,\n'
                                             '  "time" : "Wed Oct 26 15:26:28 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring-bdev_uring",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring-bdev_uring",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "8",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"spdk_bdev",\n'
                                             '        "spdk_conf" : '
                                             '"/tmp/spdk_bdev_uring_foo.conf",\n'
                                             '        "filename" : "Nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '1763398144,\n'
                                             '        "io_kbytes" : 1722068,\n'
                                             '        "bw_bytes" : 251878037,\n'
                                             '        "bw" : 245974,\n'
                                             '        "iops" : 491948.435938,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 3444131,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 100,\n'
                                             '          "max" : 20659,\n'
                                             '          "mean" : 159.195923,\n'
                                             '          "stddev" : '
                                             '134.215657,\n'
                                             '          "N" : 3444132\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 6412,\n'
                                             '          "max" : 125806,\n'
                                             '          "mean" : '
                                             '15197.289828,\n'
                                             '          "stddev" : '
                                             '4131.746276,\n'
                                             '          "N" : 3444137,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 8896,\n'
                                             '            "5.000000" : 12608,\n'
                                             '            "10.000000" : '
                                             '12992,\n'
                                             '            "20.000000" : '
                                             '13376,\n'
                                             '            "30.000000" : '
                                             '13632,\n'
                                             '            "40.000000" : '
                                             '13888,\n'
                                             '            "50.000000" : '
                                             '14144,\n'
                                             '            "60.000000" : '
                                             '14528,\n'
                                             '            "70.000000" : '
                                             '14912,\n'
                                             '            "80.000000" : '
                                             '15552,\n'
                                             '            "90.000000" : '
                                             '18816,\n'
                                             '            "95.000000" : '
                                             '22400,\n'
                                             '            "99.000000" : '
                                             '33024,\n'
                                             '            "99.500000" : '
                                             '34560,\n'
                                             '            "99.900000" : '
                                             '51456,\n'
                                             '            "99.950000" : '
                                             '69120,\n'
                                             '            "99.990000" : 80384\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 6612,\n'
                                             '          "max" : 126016,\n'
                                             '          "mean" : '
                                             '15356.485787,\n'
                                             '          "stddev" : '
                                             '4127.680316,\n'
                                             '          "N" : 3444137\n'
                                             '        },\n'
                                             '        "bw_min" : 244609,\n'
                                             '        "bw_max" : 248021,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '246103.357143,\n'
                                             '        "bw_dev" : 1102.233300,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 489218,\n'
                                             '        "iops_max" : 496042,\n'
                                             '        "iops_mean" : '
                                             '492206.785714,\n'
                                             '        "iops_stddev" : '
                                             '2204.746425,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 32.785714,\n'
                                             '      "sys_cpu" : 67.185714,\n'
                                             '      "ctx" : 12,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.100000,\n'
                                             '        "2" : 0.129118,\n'
                                             '        "4" : 72.261014,\n'
                                             '        "8" : 27.606935,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 98.422303,\n'
                                             '        "8" : 1.577697,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 1.778126,\n'
                                             '        "20" : 90.700905,\n'
                                             '        "50" : 7.415281,\n'
                                             '        "100" : 0.104874,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 8,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring-bdev_uring: (groupid=0, '
                                             'jobs=1): err= 0: pid=10800: Wed '
                                             'Oct 26 15:26:28 2022\n'
                                             '  read: IOPS=492k, BW=240MiB/s '
                                             '(252MB/s)(1682MiB/7001msec)\n'
                                             '    slat (nsec): min=100, '
                                             'max=20659, avg=159.20, '
                                             'stdev=134.22\n'
                                             '    clat (usec): min=6, max=125, '
                                             'avg=15.20, stdev= 4.13\n'
                                             '     lat (usec): min=6, max=126, '
                                             'avg=15.36, stdev= 4.13\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 8896],  '
                                             '5.00th=[12608], 10.00th=[12992], '
                                             '20.00th=[13376],\n'
                                             '     | 30.00th=[13632], '
                                             '40.00th=[13888], '
                                             '50.00th=[14144], '
                                             '60.00th=[14528],\n'
                                             '     | 70.00th=[14912], '
                                             '80.00th=[15552], '
                                             '90.00th=[18816], '
                                             '95.00th=[22400],\n'
                                             '     | 99.00th=[33024], '
                                             '99.50th=[34560], '
                                             '99.90th=[51456], '
                                             '99.95th=[69120],\n'
                                             '     | 99.99th=[80384]\n'
                                             '   bw (  KiB/s): min=244609, '
                                             'max=248021, per=100.00%, '
                                             'avg=246103.36, stdev=1102.23, '
                                             'samples=14\n'
                                             '   iops        : min=489218, '
                                             'max=496042, avg=492206.79, '
                                             'stdev=2204.75, samples=14\n'
                                             '  lat (usec)   : 10=1.78%, '
                                             '20=90.70%, 50=7.42%, 100=0.10%, '
                                             '250=0.01%\n'
                                             '  cpu          : usr=32.79%, '
                                             'sys=67.19%, ctx=12, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.1%, 2=0.1%, '
                                             '4=72.3%, 8=27.6%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=98.4%, 8=1.6%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=3444131,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=8\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=240MiB/s (252MB/s), '
                                             '240MiB/s-240MiB/s '
                                             '(252MB/s-252MB/s), io=1682MiB '
                                             '(1763MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_342.output'),
                                   'state': {'begin': 1666783698.5434163,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_8_io_uring-bdev_uring_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 1.2910566329956055,
                                             'end': 1666783699.834473,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_342.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_342.state')},
                       'cmd_343': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_512_8_io_uring-bdev_xnvme_0.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_343.output'),
                                   'state': {'begin': 1666783702.4286778,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_8_io_uring-bdev_xnvme_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 1.2081053256988525,
                                             'end': 1666783703.6367831,
                                             'err': 1,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_343.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_343.state')},
                       'cmd_344': {'output': 'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_344.output'),
                                   'state': {'begin': 1666783703.6387038,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring-bdev_xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="8" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="spdk_bdev" '
                                                    '--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" '
                                                    '--filename="Nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_8_io_uring-bdev_xnvme_0.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 12.253406524658203,
                                             'end': 1666783715.8921103,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_344.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_344.state')},
                       'cmd_345': {'output': 'io_uring-bdev_xnvme: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=spdk_bdev, iodepth=8\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666790805,\n'
                                             '  "timestamp_ms" : '
                                             '1666790805897,\n'
                                             '  "time" : "Wed Oct 26 15:26:45 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring-bdev_xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring-bdev_xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "8",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"spdk_bdev",\n'
                                             '        "spdk_conf" : '
                                             '"/tmp/spdk_bdev_xnvme_io_uring.conf",\n'
                                             '        "filename" : "Nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '1621752320,\n'
                                             '        "io_kbytes" : 1583742,\n'
                                             '        "bw_bytes" : 231645810,\n'
                                             '        "bw" : 226216,\n'
                                             '        "iops" : 452432.081131,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 3167477,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1563,\n'
                                             '          "max" : 52558,\n'
                                             '          "mean" : 1732.091699,\n'
                                             '          "stddev" : '
                                             '132.863734,\n'
                                             '          "N" : 3167478\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 250,\n'
                                             '          "max" : 126818,\n'
                                             '          "mean" : '
                                             '15699.602769,\n'
                                             '          "stddev" : '
                                             '2539.253016,\n'
                                             '          "N" : 3167484,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 14912,\n'
                                             '            "5.000000" : 14912,\n'
                                             '            "10.000000" : '
                                             '15040,\n'
                                             '            "20.000000" : '
                                             '15040,\n'
                                             '            "30.000000" : '
                                             '15168,\n'
                                             '            "40.000000" : '
                                             '15168,\n'
                                             '            "50.000000" : '
                                             '15296,\n'
                                             '            "60.000000" : '
                                             '15296,\n'
                                             '            "70.000000" : '
                                             '15296,\n'
                                             '            "80.000000" : '
                                             '15424,\n'
                                             '            "90.000000" : '
                                             '15808,\n'
                                             '            "95.000000" : '
                                             '16192,\n'
                                             '            "99.000000" : '
                                             '28288,\n'
                                             '            "99.500000" : '
                                             '30592,\n'
                                             '            "99.900000" : '
                                             '43776,\n'
                                             '            "99.950000" : '
                                             '60672,\n'
                                             '            "99.990000" : 74240\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 6412,\n'
                                             '          "max" : 128531,\n'
                                             '          "mean" : '
                                             '17431.694401,\n'
                                             '          "stddev" : '
                                             '2546.918829,\n'
                                             '          "N" : 3167484\n'
                                             '        },\n'
                                             '        "bw_min" : 224958,\n'
                                             '        "bw_max" : 227326,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '226257.285714,\n'
                                             '        "bw_dev" : 705.539339,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 449916,\n'
                                             '        "iops_max" : 454652,\n'
                                             '        "iops_mean" : '
                                             '452514.642857,\n'
                                             '        "iops_stddev" : '
                                             '1411.151936,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 30.385714,\n'
                                             '      "sys_cpu" : 69.600000,\n'
                                             '      "ctx" : 15,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 100.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 99.999984,\n'
                                             '        "8" : 0.100000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.010000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.027404,\n'
                                             '        "20" : 96.891659,\n'
                                             '        "50" : 3.001000,\n'
                                             '        "100" : 0.078675,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 8,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring-bdev_xnvme: (groupid=0, '
                                             'jobs=1): err= 0: pid=10858: Wed '
                                             'Oct 26 15:26:45 2022\n'
                                             '  read: IOPS=452k, BW=221MiB/s '
                                             '(232MB/s)(1547MiB/7001msec)\n'
                                             '    slat (nsec): min=1563, '
                                             'max=52558, avg=1732.09, '
                                             'stdev=132.86\n'
                                             '    clat (nsec): min=250, '
                                             'max=126818, avg=15699.60, '
                                             'stdev=2539.25\n'
                                             '     lat (usec): min=6, max=128, '
                                             'avg=17.43, stdev= 2.55\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[14912],  '
                                             '5.00th=[14912], 10.00th=[15040], '
                                             '20.00th=[15040],\n'
                                             '     | 30.00th=[15168], '
                                             '40.00th=[15168], '
                                             '50.00th=[15296], '
                                             '60.00th=[15296],\n'
                                             '     | 70.00th=[15296], '
                                             '80.00th=[15424], '
                                             '90.00th=[15808], '
                                             '95.00th=[16192],\n'
                                             '     | 99.00th=[28288], '
                                             '99.50th=[30592], '
                                             '99.90th=[43776], '
                                             '99.95th=[60672],\n'
                                             '     | 99.99th=[74240]\n'
                                             '   bw (  KiB/s): min=224958, '
                                             'max=227326, per=100.00%, '
                                             'avg=226257.29, stdev=705.54, '
                                             'samples=14\n'
                                             '   iops        : min=449916, '
                                             'max=454652, avg=452514.64, '
                                             'stdev=1411.15, samples=14\n'
                                             '  lat (nsec)   : 500=0.01%\n'
                                             '  lat (usec)   : 10=0.03%, '
                                             '20=96.89%, 50=3.00%, 100=0.08%, '
                                             '250=0.01%\n'
                                             '  cpu          : usr=30.39%, '
                                             'sys=69.60%, ctx=15, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, 2=0.0%, '
                                             '4=0.0%, 8=100.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.1%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=3167477,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=8\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=221MiB/s (232MB/s), '
                                             '221MiB/s-221MiB/s '
                                             '(232MB/s-232MB/s), io=1547MiB '
                                             '(1622MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_345.output'),
                                   'state': {'begin': 1666783715.8946543,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_8_io_uring-bdev_xnvme_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 1.7201216220855713,
                                             'end': 1666783717.614776,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_345.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_345.state')},
                       'cmd_346': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_512_8_io_uring-bdev_xnvme_1.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_346.output'),
                                   'state': {'begin': 1666783720.8414304,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_8_io_uring-bdev_xnvme_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 1.889601469039917,
                                             'end': 1666783722.731032,
                                             'err': 1,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_346.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_346.state')},
                       'cmd_347': {'output': 'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_347.output'),
                                   'state': {'begin': 1666783722.7332733,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring-bdev_xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="8" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="spdk_bdev" '
                                                    '--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" '
                                                    '--filename="Nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_8_io_uring-bdev_xnvme_1.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.909066915512085,
                                             'end': 1666783734.6423402,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_347.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_347.state')},
                       'cmd_348': {'output': 'io_uring-bdev_xnvme: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=spdk_bdev, iodepth=8\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666790824,\n'
                                             '  "timestamp_ms" : '
                                             '1666790824648,\n'
                                             '  "time" : "Wed Oct 26 15:27:04 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring-bdev_xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring-bdev_xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "8",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"spdk_bdev",\n'
                                             '        "spdk_conf" : '
                                             '"/tmp/spdk_bdev_xnvme_io_uring.conf",\n'
                                             '        "filename" : "Nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '1612016128,\n'
                                             '        "io_kbytes" : 1574234,\n'
                                             '        "bw_bytes" : 230255124,\n'
                                             '        "bw" : 224858,\n'
                                             '        "iops" : 449715.897729,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 3148461,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1573,\n'
                                             '          "max" : 40045,\n'
                                             '          "mean" : 1761.557793,\n'
                                             '          "stddev" : '
                                             '123.973748,\n'
                                             '          "N" : 3148462\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 250,\n'
                                             '          "max" : 120826,\n'
                                             '          "mean" : '
                                             '15794.247972,\n'
                                             '          "stddev" : '
                                             '2581.682894,\n'
                                             '          "N" : 3148468,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 14912,\n'
                                             '            "5.000000" : 15040,\n'
                                             '            "10.000000" : '
                                             '15040,\n'
                                             '            "20.000000" : '
                                             '15168,\n'
                                             '            "30.000000" : '
                                             '15168,\n'
                                             '            "40.000000" : '
                                             '15296,\n'
                                             '            "50.000000" : '
                                             '15296,\n'
                                             '            "60.000000" : '
                                             '15424,\n'
                                             '            "70.000000" : '
                                             '15424,\n'
                                             '            "80.000000" : '
                                             '15808,\n'
                                             '            "90.000000" : '
                                             '16064,\n'
                                             '            "95.000000" : '
                                             '16192,\n'
                                             '            "99.000000" : '
                                             '28032,\n'
                                             '            "99.500000" : '
                                             '30592,\n'
                                             '            "99.900000" : '
                                             '49408,\n'
                                             '            "99.950000" : '
                                             '64768,\n'
                                             '            "99.990000" : 75264\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 5831,\n'
                                             '          "max" : 122500,\n'
                                             '          "mean" : '
                                             '17555.805718,\n'
                                             '          "stddev" : '
                                             '2589.532167,\n'
                                             '          "N" : 3148468\n'
                                             '        },\n'
                                             '        "bw_min" : 224054,\n'
                                             '        "bw_max" : 227215,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '224976.142857,\n'
                                             '        "bw_dev" : 840.788902,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 448108,\n'
                                             '        "iops_max" : 454430,\n'
                                             '        "iops_mean" : '
                                             '449952.428571,\n'
                                             '        "iops_stddev" : '
                                             '1681.479647,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 31.585714,\n'
                                             '      "sys_cpu" : 68.400000,\n'
                                             '      "ctx" : 11,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 100.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 99.999984,\n'
                                             '        "8" : 0.100000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.010000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.032619,\n'
                                             '        "20" : 96.921004,\n'
                                             '        "50" : 2.948266,\n'
                                             '        "100" : 0.097254,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 8,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring-bdev_xnvme: (groupid=0, '
                                             'jobs=1): err= 0: pid=10915: Wed '
                                             'Oct 26 15:27:04 2022\n'
                                             '  read: IOPS=450k, BW=220MiB/s '
                                             '(230MB/s)(1537MiB/7001msec)\n'
                                             '    slat (nsec): min=1573, '
                                             'max=40045, avg=1761.56, '
                                             'stdev=123.97\n'
                                             '    clat (nsec): min=250, '
                                             'max=120826, avg=15794.25, '
                                             'stdev=2581.68\n'
                                             '     lat (usec): min=5, max=122, '
                                             'avg=17.56, stdev= 2.59\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[14912],  '
                                             '5.00th=[15040], 10.00th=[15040], '
                                             '20.00th=[15168],\n'
                                             '     | 30.00th=[15168], '
                                             '40.00th=[15296], '
                                             '50.00th=[15296], '
                                             '60.00th=[15424],\n'
                                             '     | 70.00th=[15424], '
                                             '80.00th=[15808], '
                                             '90.00th=[16064], '
                                             '95.00th=[16192],\n'
                                             '     | 99.00th=[28032], '
                                             '99.50th=[30592], '
                                             '99.90th=[49408], '
                                             '99.95th=[64768],\n'
                                             '     | 99.99th=[75264]\n'
                                             '   bw (  KiB/s): min=224054, '
                                             'max=227215, per=100.00%, '
                                             'avg=224976.14, stdev=840.79, '
                                             'samples=14\n'
                                             '   iops        : min=448108, '
                                             'max=454430, avg=449952.43, '
                                             'stdev=1681.48, samples=14\n'
                                             '  lat (nsec)   : 500=0.01%\n'
                                             '  lat (usec)   : 10=0.03%, '
                                             '20=96.92%, 50=2.95%, 100=0.10%, '
                                             '250=0.01%\n'
                                             '  cpu          : usr=31.59%, '
                                             'sys=68.40%, ctx=11, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, 2=0.0%, '
                                             '4=0.0%, 8=100.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.1%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=3148461,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=8\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=220MiB/s (230MB/s), '
                                             '220MiB/s-220MiB/s '
                                             '(230MB/s-230MB/s), io=1537MiB '
                                             '(1612MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_348.output'),
                                   'state': {'begin': 1666783734.6453257,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_8_io_uring-bdev_xnvme_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 1.0389409065246582,
                                             'end': 1666783735.6842666,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_348.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_348.state')},
                       'cmd_349': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_512_8_io_uring-bdev_xnvme_2.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_349.output'),
                                   'state': {'begin': 1666783738.3035614,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_8_io_uring-bdev_xnvme_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 1.1200616359710693,
                                             'end': 1666783739.423623,
                                             'err': 1,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_349.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_349.state')},
                       'cmd_35': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_35.output'),
                                  'state': {'begin': 1666782203.7398446,
                                            'cmd': '/opt/fio/bin/fio '
                                                   '--name="io_uring-reference" '
                                                   '--rw="randread" '
                                                   '--size="12G" --iodepth="1" '
                                                   '--bs="512" --direct="1" '
                                                   '--thread="1" '
                                                   '--time_based="1" '
                                                   '--runtime="7" '
                                                   '--ramp_time="3" '
                                                   '--norandommap="1" '
                                                   '--allow_file_create="0" '
                                                   '--ioengine="io_uring" '
                                                   '--filename="/dev/nvme0n1" '
                                                   '--output-format="normal,json" '
                                                   '--output="/tmp/fio-output_512_1_io_uring-reference_2.txt" '
                                                   '--cpus_allowed="1"',
                                            'cwd': 'None',
                                            'elapsed': 11.05388355255127,
                                            'end': 1666782214.793728,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_35.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_35.state')},
                       'cmd_350': {'output': 'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_350.output'),
                                   'state': {'begin': 1666783739.425886,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring-bdev_xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="8" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="spdk_bdev" '
                                                    '--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" '
                                                    '--filename="Nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_8_io_uring-bdev_xnvme_2.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.944929599761963,
                                             'end': 1666783751.3708155,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_350.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_350.state')},
                       'cmd_351': {'output': 'io_uring-bdev_xnvme: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=spdk_bdev, iodepth=8\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666790841,\n'
                                             '  "timestamp_ms" : '
                                             '1666790841375,\n'
                                             '  "time" : "Wed Oct 26 15:27:21 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring-bdev_xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring-bdev_xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "8",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"spdk_bdev",\n'
                                             '        "spdk_conf" : '
                                             '"/tmp/spdk_bdev_xnvme_io_uring.conf",\n'
                                             '        "filename" : "Nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '1616416768,\n'
                                             '        "io_kbytes" : 1578532,\n'
                                             '        "bw_bytes" : 230883697,\n'
                                             '        "bw" : 225472,\n'
                                             '        "iops" : 450943.579489,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 3157056,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1572,\n'
                                             '          "max" : 17793,\n'
                                             '          "mean" : 1750.871442,\n'
                                             '          "stddev" : '
                                             '124.108060,\n'
                                             '          "N" : 3157057\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 260,\n'
                                             '          "max" : 128782,\n'
                                             '          "mean" : '
                                             '15750.550326,\n'
                                             '          "stddev" : '
                                             '2540.179660,\n'
                                             '          "N" : 3157063,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 14912,\n'
                                             '            "5.000000" : 15040,\n'
                                             '            "10.000000" : '
                                             '15040,\n'
                                             '            "20.000000" : '
                                             '15168,\n'
                                             '            "30.000000" : '
                                             '15168,\n'
                                             '            "40.000000" : '
                                             '15168,\n'
                                             '            "50.000000" : '
                                             '15296,\n'
                                             '            "60.000000" : '
                                             '15296,\n'
                                             '            "70.000000" : '
                                             '15424,\n'
                                             '            "80.000000" : '
                                             '15808,\n'
                                             '            "90.000000" : '
                                             '15936,\n'
                                             '            "95.000000" : '
                                             '16192,\n'
                                             '            "99.000000" : '
                                             '28288,\n'
                                             '            "99.500000" : '
                                             '30592,\n'
                                             '            "99.900000" : '
                                             '44288,\n'
                                             '            "99.950000" : '
                                             '63232,\n'
                                             '            "99.990000" : 74240\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 6081,\n'
                                             '          "max" : 130635,\n'
                                             '          "mean" : '
                                             '17501.421643,\n'
                                             '          "stddev" : '
                                             '2547.987942,\n'
                                             '          "N" : 3157063\n'
                                             '        },\n'
                                             '        "bw_min" : 224516,\n'
                                             '        "bw_max" : 226843,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '225679.230769,\n'
                                             '        "bw_dev" : 917.421582,\n'
                                             '        "bw_samples" : 13,\n'
                                             '        "iops_min" : 449032,\n'
                                             '        "iops_max" : 453686,\n'
                                             '        "iops_mean" : '
                                             '451358.615385,\n'
                                             '        "iops_stddev" : '
                                             '1835.033630,\n'
                                             '        "iops_samples" : 13\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 31.128571,\n'
                                             '      "sys_cpu" : 68.857143,\n'
                                             '      "ctx" : 8,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 100.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 99.999984,\n'
                                             '        "8" : 0.100000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.010000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 0.031517,\n'
                                             '        "20" : 96.933409,\n'
                                             '        "50" : 2.949868,\n'
                                             '        "100" : 0.084382,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 8,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring-bdev_xnvme: (groupid=0, '
                                             'jobs=1): err= 0: pid=10974: Wed '
                                             'Oct 26 15:27:21 2022\n'
                                             '  read: IOPS=451k, BW=220MiB/s '
                                             '(231MB/s)(1542MiB/7001msec)\n'
                                             '    slat (nsec): min=1572, '
                                             'max=17793, avg=1750.87, '
                                             'stdev=124.11\n'
                                             '    clat (nsec): min=260, '
                                             'max=128782, avg=15750.55, '
                                             'stdev=2540.18\n'
                                             '     lat (usec): min=6, max=130, '
                                             'avg=17.50, stdev= 2.55\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[14912],  '
                                             '5.00th=[15040], 10.00th=[15040], '
                                             '20.00th=[15168],\n'
                                             '     | 30.00th=[15168], '
                                             '40.00th=[15168], '
                                             '50.00th=[15296], '
                                             '60.00th=[15296],\n'
                                             '     | 70.00th=[15424], '
                                             '80.00th=[15808], '
                                             '90.00th=[15936], '
                                             '95.00th=[16192],\n'
                                             '     | 99.00th=[28288], '
                                             '99.50th=[30592], '
                                             '99.90th=[44288], '
                                             '99.95th=[63232],\n'
                                             '     | 99.99th=[74240]\n'
                                             '   bw (  KiB/s): min=224516, '
                                             'max=226843, per=100.00%, '
                                             'avg=225679.23, stdev=917.42, '
                                             'samples=13\n'
                                             '   iops        : min=449032, '
                                             'max=453686, avg=451358.62, '
                                             'stdev=1835.03, samples=13\n'
                                             '  lat (nsec)   : 500=0.01%\n'
                                             '  lat (usec)   : 4=0.01%, '
                                             '10=0.03%, 20=96.93%, 50=2.95%, '
                                             '100=0.08%\n'
                                             '  lat (usec)   : 250=0.01%\n'
                                             '  cpu          : usr=31.13%, '
                                             'sys=68.86%, ctx=8, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, 2=0.0%, '
                                             '4=0.0%, 8=100.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.1%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=3157056,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=8\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=220MiB/s (231MB/s), '
                                             '220MiB/s-220MiB/s '
                                             '(231MB/s-231MB/s), io=1542MiB '
                                             '(1616MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_351.output'),
                                   'state': {'begin': 1666783751.3734238,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_8_io_uring-bdev_xnvme_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 1.1073675155639648,
                                             'end': 1666783752.4807913,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_351.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_351.state')},
                       'cmd_352': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_512_8_io_uring-xnvme_0.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_352.output'),
                                   'state': {'begin': 1666783753.932052,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_8_io_uring-xnvme_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 1.4276745319366455,
                                             'end': 1666783755.3597264,
                                             'err': 1,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_352.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_352.state')},
                       'cmd_353': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_353.output'),
                                   'state': {'begin': 1666783755.3624756,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring-xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="8" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
                                                    '--xnvme_be="linux" '
                                                    '--xnvme_async="io_uring" '
                                                    '--xnvme_sync="nvme" '
                                                    '--xnvme_admin="nvme" '
                                                    '--xnvme_dev_nsid="1" '
                                                    '--filename="/dev/nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_8_io_uring-xnvme_0.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.359984636306763,
                                             'end': 1666783766.7224603,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_353.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_353.state')},
                       'cmd_354': {'output': 'io_uring-xnvme: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=xnvme, iodepth=8\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666790856,\n'
                                             '  "timestamp_ms" : '
                                             '1666790856790,\n'
                                             '  "time" : "Wed Oct 26 15:27:36 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring-xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring-xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "8",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
                                             '        "xnvme_be" : "linux",\n'
                                             '        "xnvme_async" : '
                                             '"io_uring",\n'
                                             '        "xnvme_sync" : "nvme",\n'
                                             '        "xnvme_admin" : "nvme",\n'
                                             '        "xnvme_dev_nsid" : "1",\n'
                                             '        "filename" : '
                                             '"/dev/nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '1419834368,\n'
                                             '        "io_kbytes" : 1386557,\n'
                                             '        "bw_bytes" : 202804509,\n'
                                             '        "bw" : 198051,\n'
                                             '        "iops" : 396101.414084,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 2773106,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1393,\n'
                                             '          "max" : 70382,\n'
                                             '          "mean" : 2032.974013,\n'
                                             '          "stddev" : '
                                             '315.234496,\n'
                                             '          "N" : 2773107\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 4167,\n'
                                             '          "max" : 134191,\n'
                                             '          "mean" : '
                                             '17867.042998,\n'
                                             '          "stddev" : '
                                             '3010.820945,\n'
                                             '          "N" : 2773113,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 13376,\n'
                                             '            "5.000000" : 15424,\n'
                                             '            "10.000000" : '
                                             '16768,\n'
                                             '            "20.000000" : '
                                             '17024,\n'
                                             '            "30.000000" : '
                                             '17280,\n'
                                             '            "40.000000" : '
                                             '17536,\n'
                                             '            "50.000000" : '
                                             '17536,\n'
                                             '            "60.000000" : '
                                             '17536,\n'
                                             '            "70.000000" : '
                                             '18048,\n'
                                             '            "80.000000" : '
                                             '18048,\n'
                                             '            "90.000000" : '
                                             '18304,\n'
                                             '            "95.000000" : '
                                             '19584,\n'
                                             '            "99.000000" : '
                                             '31616,\n'
                                             '            "99.500000" : '
                                             '34560,\n'
                                             '            "99.900000" : '
                                             '50432,\n'
                                             '            "99.950000" : '
                                             '64256,\n'
                                             '            "99.990000" : 77312\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 6372,\n'
                                             '          "max" : 135744,\n'
                                             '          "mean" : '
                                             '19900.015943,\n'
                                             '          "stddev" : '
                                             '2965.963506,\n'
                                             '          "N" : 2773113\n'
                                             '        },\n'
                                             '        "bw_min" : 197423,\n'
                                             '        "bw_max" : 198917,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '198149.500000,\n'
                                             '        "bw_dev" : 471.310897,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 394847,\n'
                                             '        "iops_max" : 397834,\n'
                                             '        "iops_mean" : '
                                             '396299.142857,\n'
                                             '        "iops_stddev" : '
                                             '942.487532,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 20.971429,\n'
                                             '      "sys_cpu" : 76.928571,\n'
                                             '      "ctx" : 16494,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 1,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 100.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 99.999982,\n'
                                             '        "8" : 0.100000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.102989,\n'
                                             '        "20" : 95.387482,\n'
                                             '        "50" : 4.409135,\n'
                                             '        "100" : 0.099780,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 8,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring-xnvme: (groupid=0, '
                                             'jobs=1): err= 0: pid=11022: Wed '
                                             'Oct 26 15:27:36 2022\n'
                                             '  read: IOPS=396k, BW=193MiB/s '
                                             '(203MB/s)(1354MiB/7001msec)\n'
                                             '    slat (nsec): min=1393, '
                                             'max=70382, avg=2032.97, '
                                             'stdev=315.23\n'
                                             '    clat (usec): min=4, max=134, '
                                             'avg=17.87, stdev= 3.01\n'
                                             '     lat (usec): min=6, max=135, '
                                             'avg=19.90, stdev= 2.97\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[13376],  '
                                             '5.00th=[15424], 10.00th=[16768], '
                                             '20.00th=[17024],\n'
                                             '     | 30.00th=[17280], '
                                             '40.00th=[17536], '
                                             '50.00th=[17536], '
                                             '60.00th=[17536],\n'
                                             '     | 70.00th=[18048], '
                                             '80.00th=[18048], '
                                             '90.00th=[18304], '
                                             '95.00th=[19584],\n'
                                             '     | 99.00th=[31616], '
                                             '99.50th=[34560], '
                                             '99.90th=[50432], '
                                             '99.95th=[64256],\n'
                                             '     | 99.99th=[77312]\n'
                                             '   bw (  KiB/s): min=197423, '
                                             'max=198917, per=100.00%, '
                                             'avg=198149.50, stdev=471.31, '
                                             'samples=14\n'
                                             '   iops        : min=394847, '
                                             'max=397834, avg=396299.14, '
                                             'stdev=942.49, samples=14\n'
                                             '  lat (usec)   : 10=0.10%, '
                                             '20=95.39%, 50=4.41%, 100=0.10%, '
                                             '250=0.01%\n'
                                             '  cpu          : usr=20.97%, '
                                             'sys=76.93%, ctx=16494, majf=0, '
                                             'minf=1\n'
                                             '  IO depths    : 1=0.0%, 2=0.0%, '
                                             '4=0.0%, 8=100.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.1%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=2773106,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=8\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=193MiB/s (203MB/s), '
                                             '193MiB/s-193MiB/s '
                                             '(203MB/s-203MB/s), io=1354MiB '
                                             '(1420MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_354.output'),
                                   'state': {'begin': 1666783766.7253551,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_8_io_uring-xnvme_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 1.3404881954193115,
                                             'end': 1666783768.0658433,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_354.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_354.state')},
                       'cmd_355': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_512_8_io_uring-xnvme_1.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_355.output'),
                                   'state': {'begin': 1666783769.5046735,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_8_io_uring-xnvme_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 1.3336870670318604,
                                             'end': 1666783770.8383605,
                                             'err': 1,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_355.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_355.state')},
                       'cmd_356': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_356.output'),
                                   'state': {'begin': 1666783770.8401787,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring-xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="8" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
                                                    '--xnvme_be="linux" '
                                                    '--xnvme_async="io_uring" '
                                                    '--xnvme_sync="nvme" '
                                                    '--xnvme_admin="nvme" '
                                                    '--xnvme_dev_nsid="1" '
                                                    '--filename="/dev/nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_8_io_uring-xnvme_1.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 12.943699359893799,
                                             'end': 1666783783.783878,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_356.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_356.state')},
                       'cmd_357': {'output': 'io_uring-xnvme: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=xnvme, iodepth=8\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666790873,\n'
                                             '  "timestamp_ms" : '
                                             '1666790873848,\n'
                                             '  "time" : "Wed Oct 26 15:27:53 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring-xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring-xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "8",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
                                             '        "xnvme_be" : "linux",\n'
                                             '        "xnvme_async" : '
                                             '"io_uring",\n'
                                             '        "xnvme_sync" : "nvme",\n'
                                             '        "xnvme_admin" : "nvme",\n'
                                             '        "xnvme_dev_nsid" : "1",\n'
                                             '        "filename" : '
                                             '"/dev/nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '1414161408,\n'
                                             '        "io_kbytes" : 1381017,\n'
                                             '        "bw_bytes" : 201994201,\n'
                                             '        "bw" : 197259,\n'
                                             '        "iops" : 394518.925868,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 2762027,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1392,\n'
                                             '          "max" : 30357,\n'
                                             '          "mean" : 2037.367052,\n'
                                             '          "stddev" : '
                                             '294.648162,\n'
                                             '          "N" : 2762027\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 4098,\n'
                                             '          "max" : 135283,\n'
                                             '          "mean" : '
                                             '17936.435806,\n'
                                             '          "stddev" : '
                                             '3001.136611,\n'
                                             '          "N" : 2762034,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 13376,\n'
                                             '            "5.000000" : 15552,\n'
                                             '            "10.000000" : '
                                             '16768,\n'
                                             '            "20.000000" : '
                                             '17280,\n'
                                             '            "30.000000" : '
                                             '17536,\n'
                                             '            "40.000000" : '
                                             '17536,\n'
                                             '            "50.000000" : '
                                             '17536,\n'
                                             '            "60.000000" : '
                                             '17536,\n'
                                             '            "70.000000" : '
                                             '18048,\n'
                                             '            "80.000000" : '
                                             '18304,\n'
                                             '            "90.000000" : '
                                             '18304,\n'
                                             '            "95.000000" : '
                                             '19584,\n'
                                             '            "99.000000" : '
                                             '31616,\n'
                                             '            "99.500000" : '
                                             '35072,\n'
                                             '            "99.900000" : '
                                             '51456,\n'
                                             '            "99.950000" : '
                                             '66048,\n'
                                             '            "99.990000" : 76288\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 7033,\n'
                                             '          "max" : 136786,\n'
                                             '          "mean" : '
                                             '19973.801927,\n'
                                             '          "stddev" : '
                                             '2955.591710,\n'
                                             '          "N" : 2762034\n'
                                             '        },\n'
                                             '        "bw_min" : 196032,\n'
                                             '        "bw_max" : 198923,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '197370.428571,\n'
                                             '        "bw_dev" : 916.608608,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 392064,\n'
                                             '        "iops_max" : 397846,\n'
                                             '        "iops_mean" : '
                                             '394741.000000,\n'
                                             '        "iops_stddev" : '
                                             '1833.052687,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 20.842857,\n'
                                             '      "sys_cpu" : 77.085714,\n'
                                             '      "ctx" : 16725,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 100.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 99.999982,\n'
                                             '        "8" : 0.100000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.081208,\n'
                                             '        "20" : 95.414129,\n'
                                             '        "50" : 4.391956,\n'
                                             '        "100" : 0.112671,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 8,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring-xnvme: (groupid=0, '
                                             'jobs=1): err= 0: pid=11070: Wed '
                                             'Oct 26 15:27:53 2022\n'
                                             '  read: IOPS=395k, BW=193MiB/s '
                                             '(202MB/s)(1349MiB/7001msec)\n'
                                             '    slat (nsec): min=1392, '
                                             'max=30357, avg=2037.37, '
                                             'stdev=294.65\n'
                                             '    clat (usec): min=4, max=135, '
                                             'avg=17.94, stdev= 3.00\n'
                                             '     lat (usec): min=7, max=136, '
                                             'avg=19.97, stdev= 2.96\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[13376],  '
                                             '5.00th=[15552], 10.00th=[16768], '
                                             '20.00th=[17280],\n'
                                             '     | 30.00th=[17536], '
                                             '40.00th=[17536], '
                                             '50.00th=[17536], '
                                             '60.00th=[17536],\n'
                                             '     | 70.00th=[18048], '
                                             '80.00th=[18304], '
                                             '90.00th=[18304], '
                                             '95.00th=[19584],\n'
                                             '     | 99.00th=[31616], '
                                             '99.50th=[35072], '
                                             '99.90th=[51456], '
                                             '99.95th=[66048],\n'
                                             '     | 99.99th=[76288]\n'
                                             '   bw (  KiB/s): min=196032, '
                                             'max=198923, per=100.00%, '
                                             'avg=197370.43, stdev=916.61, '
                                             'samples=14\n'
                                             '   iops        : min=392064, '
                                             'max=397846, avg=394741.00, '
                                             'stdev=1833.05, samples=14\n'
                                             '  lat (usec)   : 10=0.08%, '
                                             '20=95.41%, 50=4.39%, 100=0.11%, '
                                             '250=0.01%\n'
                                             '  cpu          : usr=20.84%, '
                                             'sys=77.09%, ctx=16725, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, 2=0.0%, '
                                             '4=0.0%, 8=100.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.1%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=2762027,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=8\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=193MiB/s (202MB/s), '
                                             '193MiB/s-193MiB/s '
                                             '(202MB/s-202MB/s), io=1349MiB '
                                             '(1414MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_357.output'),
                                   'state': {'begin': 1666783783.7866464,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_8_io_uring-xnvme_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 1.2279436588287354,
                                             'end': 1666783785.01459,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_357.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_357.state')},
                       'cmd_358': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_512_8_io_uring-xnvme_2.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_358.output'),
                                   'state': {'begin': 1666783788.3961504,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_8_io_uring-xnvme_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 1.418452501296997,
                                             'end': 1666783789.8146029,
                                             'err': 1,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_358.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_358.state')},
                       'cmd_359': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_359.output'),
                                   'state': {'begin': 1666783789.81669,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring-xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="8" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
                                                    '--xnvme_be="linux" '
                                                    '--xnvme_async="io_uring" '
                                                    '--xnvme_sync="nvme" '
                                                    '--xnvme_admin="nvme" '
                                                    '--xnvme_dev_nsid="1" '
                                                    '--filename="/dev/nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_8_io_uring-xnvme_2.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.817346572875977,
                                             'end': 1666783801.6340365,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_359.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_359.state')},
                       'cmd_36': {'output': 'io_uring-reference: (g=0): '
                                            'rw=randread, bs=(R) 512B-512B, '
                                            '(W) 512B-512B, (T) 512B-512B, '
                                            'ioengine=io_uring, iodepth=1\n'
                                            'fio-3.32-dirty\n'
                                            'Starting 1 thread\n'
                                            '{\n'
                                            '  "fio version" : '
                                            '"fio-3.32-dirty",\n'
                                            '  "timestamp" : 1666789304,\n'
                                            '  "timestamp_ms" : '
                                            '1666789304886,\n'
                                            '  "time" : "Wed Oct 26 15:01:44 '
                                            '2022",\n'
                                            '  "jobs" : [\n'
                                            '    {\n'
                                            '      "jobname" : '
                                            '"io_uring-reference",\n'
                                            '      "groupid" : 0,\n'
                                            '      "error" : 0,\n'
                                            '      "eta" : 0,\n'
                                            '      "elapsed" : 11,\n'
                                            '      "job options" : {\n'
                                            '        "name" : '
                                            '"io_uring-reference",\n'
                                            '        "rw" : "randread",\n'
                                            '        "size" : "12G",\n'
                                            '        "iodepth" : "1",\n'
                                            '        "bs" : "512",\n'
                                            '        "direct" : "1",\n'
                                            '        "thread" : "1",\n'
                                            '        "time_based" : "1",\n'
                                            '        "runtime" : "7",\n'
                                            '        "ramp_time" : "3",\n'
                                            '        "norandommap" : "1",\n'
                                            '        "allow_file_create" : '
                                            '"0",\n'
                                            '        "ioengine" : "io_uring",\n'
                                            '        "filename" : '
                                            '"/dev/nvme0n1",\n'
                                            '        "cpus_allowed" : "1"\n'
                                            '      },\n'
                                            '      "read" : {\n'
                                            '        "io_bytes" : 393098240,\n'
                                            '        "io_kbytes" : 383885,\n'
                                            '        "bw_bytes" : 56148870,\n'
                                            '        "bw" : 54832,\n'
                                            '        "iops" : 109665.762034,\n'
                                            '        "runtime" : 7001,\n'
                                            '        "total_ios" : 767770,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 1493,\n'
                                            '          "max" : 6332,\n'
                                            '          "mean" : 1588.558791,\n'
                                            '          "stddev" : 74.080772,\n'
                                            '          "N" : 767770\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 2945,\n'
                                            '          "max" : 114725,\n'
                                            '          "mean" : 7224.469301,\n'
                                            '          "stddev" : 999.991459,\n'
                                            '          "N" : 767770,\n'
                                            '          "percentile" : {\n'
                                            '            "1.000000" : 6752,\n'
                                            '            "5.000000" : 6944,\n'
                                            '            "10.000000" : 6944,\n'
                                            '            "20.000000" : 7008,\n'
                                            '            "30.000000" : 7008,\n'
                                            '            "40.000000" : 7072,\n'
                                            '            "50.000000" : 7136,\n'
                                            '            "60.000000" : 7136,\n'
                                            '            "70.000000" : 7200,\n'
                                            '            "80.000000" : 7200,\n'
                                            '            "90.000000" : 7328,\n'
                                            '            "95.000000" : 7456,\n'
                                            '            "99.000000" : 10432,\n'
                                            '            "99.500000" : 13632,\n'
                                            '            "99.900000" : 15552,\n'
                                            '            "99.950000" : 15808,\n'
                                            '            "99.990000" : 59136\n'
                                            '          }\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 7434,\n'
                                            '          "max" : 116268,\n'
                                            '          "mean" : 8813.028092,\n'
                                            '          "stddev" : '
                                            '1004.849320,\n'
                                            '          "N" : 767770\n'
                                            '        },\n'
                                            '        "bw_min" : 54684,\n'
                                            '        "bw_max" : 55007,\n'
                                            '        "bw_agg" : 100.000000,\n'
                                            '        "bw_mean" : '
                                            '54870.642857,\n'
                                            '        "bw_dev" : 100.171090,\n'
                                            '        "bw_samples" : 14,\n'
                                            '        "iops_min" : 109368,\n'
                                            '        "iops_max" : 110014,\n'
                                            '        "iops_mean" : '
                                            '109741.500000,\n'
                                            '        "iops_stddev" : '
                                            '200.451701,\n'
                                            '        "iops_samples" : 14\n'
                                            '      },\n'
                                            '      "write" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "trim" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "sync" : {\n'
                                            '        "total_ios" : 0,\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        }\n'
                                            '      },\n'
                                            '      "job_runtime" : 7000,\n'
                                            '      "usr_cpu" : 7.971429,\n'
                                            '      "sys_cpu" : 35.742857,\n'
                                            '      "ctx" : 767772,\n'
                                            '      "majf" : 0,\n'
                                            '      "minf" : 0,\n'
                                            '      "iodepth_level" : {\n'
                                            '        "1" : 100.000000,\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_submit" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_complete" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ns" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_us" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.010000,\n'
                                            '        "10" : 98.707556,\n'
                                            '        "20" : 1.272517,\n'
                                            '        "50" : 0.010000,\n'
                                            '        "100" : 0.014588,\n'
                                            '        "250" : 0.010000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ms" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000,\n'
                                            '        "2000" : 0.000000,\n'
                                            '        ">=2000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_depth" : 1,\n'
                                            '      "latency_target" : 0,\n'
                                            '      "latency_percentile" : '
                                            '100.000000,\n'
                                            '      "latency_window" : 0\n'
                                            '    }\n'
                                            '  ],\n'
                                            '  "disk_util" : [\n'
                                            '    {\n'
                                            '      "name" : "nvme0n1",\n'
                                            '      "read_ios" : 1100470,\n'
                                            '      "write_ios" : 0,\n'
                                            '      "read_merges" : 0,\n'
                                            '      "write_merges" : 0,\n'
                                            '      "read_ticks" : 5516,\n'
                                            '      "write_ticks" : 0,\n'
                                            '      "in_queue" : 5516,\n'
                                            '      "util" : 99.108381\n'
                                            '    }\n'
                                            '  ]\n'
                                            '}\n'
                                            '\n'
                                            'io_uring-reference: (groupid=0, '
                                            'jobs=1): err= 0: pid=5483: Wed '
                                            'Oct 26 15:01:44 2022\n'
                                            '  read: IOPS=110k, BW=53.5MiB/s '
                                            '(56.1MB/s)(375MiB/7001msec)\n'
                                            '    slat (nsec): min=1493, '
                                            'max=6332, avg=1588.56, '
                                            'stdev=74.08\n'
                                            '    clat (nsec): min=2945, '
                                            'max=114725, avg=7224.47, '
                                            'stdev=999.99\n'
                                            '     lat (usec): min=7, max=116, '
                                            'avg= 8.81, stdev= 1.00\n'
                                            '    clat percentiles (nsec):\n'
                                            '     |  1.00th=[ 6752],  5.00th=[ '
                                            '6944], 10.00th=[ 6944], 20.00th=[ '
                                            '7008],\n'
                                            '     | 30.00th=[ 7008], 40.00th=[ '
                                            '7072], 50.00th=[ 7136], 60.00th=[ '
                                            '7136],\n'
                                            '     | 70.00th=[ 7200], 80.00th=[ '
                                            '7200], 90.00th=[ 7328], 95.00th=[ '
                                            '7456],\n'
                                            '     | 99.00th=[10432], '
                                            '99.50th=[13632], 99.90th=[15552], '
                                            '99.95th=[15808],\n'
                                            '     | 99.99th=[59136]\n'
                                            '   bw (  KiB/s): min=54684, '
                                            'max=55007, per=100.00%, '
                                            'avg=54870.64, stdev=100.17, '
                                            'samples=14\n'
                                            '   iops        : min=109368, '
                                            'max=110014, avg=109741.50, '
                                            'stdev=200.45, samples=14\n'
                                            '  lat (usec)   : 4=0.01%, '
                                            '10=98.71%, 20=1.27%, 50=0.01%, '
                                            '100=0.01%\n'
                                            '  lat (usec)   : 250=0.01%\n'
                                            '  cpu          : usr=7.97%, '
                                            'sys=35.74%, ctx=767772, majf=0, '
                                            'minf=0\n'
                                            '  IO depths    : 1=100.0%, '
                                            '2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, >=64=0.0%\n'
                                            '     submit    : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     complete  : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     issued rwts: '
                                            'total=767770,0,0,0 short=0,0,0,0 '
                                            'dropped=0,0,0,0\n'
                                            '     latency   : target=0, '
                                            'window=0, percentile=100.00%, '
                                            'depth=1\n'
                                            '\n'
                                            'Run status group 0 (all jobs):\n'
                                            '   READ: bw=53.5MiB/s (56.1MB/s), '
                                            '53.5MiB/s-53.5MiB/s '
                                            '(56.1MB/s-56.1MB/s), io=375MiB '
                                            '(393MB), run=7001-7001msec\n'
                                            '\n'
                                            'Disk stats (read/write):\n'
                                            '  nvme0n1: ios=1100470/0, '
                                            'merge=0/0, ticks=5516/0, '
                                            'in_queue=5516, util=99.11%\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_36.output'),
                                  'state': {'begin': 1666782214.7961264,
                                            'cmd': 'cat '
                                                   '/tmp/fio-output_512_1_io_uring-reference_2.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.8309309482574463,
                                            'end': 1666782215.6270573,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_36.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_36.state')},
                       'cmd_360': {'output': 'io_uring-xnvme: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=xnvme, iodepth=8\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666790891,\n'
                                             '  "timestamp_ms" : '
                                             '1666790891701,\n'
                                             '  "time" : "Wed Oct 26 15:28:11 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring-xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring-xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "8",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
                                             '        "xnvme_be" : "linux",\n'
                                             '        "xnvme_async" : '
                                             '"io_uring",\n'
                                             '        "xnvme_sync" : "nvme",\n'
                                             '        "xnvme_admin" : "nvme",\n'
                                             '        "xnvme_dev_nsid" : "1",\n'
                                             '        "filename" : '
                                             '"/dev/nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '1422672384,\n'
                                             '        "io_kbytes" : 1389328,\n'
                                             '        "bw_bytes" : 203209882,\n'
                                             '        "bw" : 198447,\n'
                                             '        "iops" : 396893.158120,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 2778649,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1402,\n'
                                             '          "max" : 29996,\n'
                                             '          "mean" : 2024.455626,\n'
                                             '          "stddev" : '
                                             '305.377303,\n'
                                             '          "N" : 2778650\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 2254,\n'
                                             '          "max" : 131687,\n'
                                             '          "mean" : '
                                             '17831.837688,\n'
                                             '          "stddev" : '
                                             '3036.634295,\n'
                                             '          "N" : 2778656,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 13248,\n'
                                             '            "5.000000" : 15168,\n'
                                             '            "10.000000" : '
                                             '16512,\n'
                                             '            "20.000000" : '
                                             '17024,\n'
                                             '            "30.000000" : '
                                             '17280,\n'
                                             '            "40.000000" : '
                                             '17536,\n'
                                             '            "50.000000" : '
                                             '17536,\n'
                                             '            "60.000000" : '
                                             '17536,\n'
                                             '            "70.000000" : '
                                             '17536,\n'
                                             '            "80.000000" : '
                                             '18048,\n'
                                             '            "90.000000" : '
                                             '18304,\n'
                                             '            "95.000000" : '
                                             '19584,\n'
                                             '            "99.000000" : '
                                             '31872,\n'
                                             '            "99.500000" : '
                                             '35072,\n'
                                             '            "99.900000" : '
                                             '50944,\n'
                                             '            "99.950000" : '
                                             '64768,\n'
                                             '            "99.990000" : 77312\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 6422,\n'
                                             '          "max" : 133160,\n'
                                             '          "mean" : '
                                             '19856.292749,\n'
                                             '          "stddev" : '
                                             '2987.817199,\n'
                                             '          "N" : 2778656\n'
                                             '        },\n'
                                             '        "bw_min" : 197185,\n'
                                             '        "bw_max" : 199877,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '198482.571429,\n'
                                             '        "bw_dev" : 752.010404,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 394370,\n'
                                             '        "iops_max" : 399754,\n'
                                             '        "iops_mean" : '
                                             '396965.142857,\n'
                                             '        "iops_stddev" : '
                                             '1504.020809,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 22.071429,\n'
                                             '      "sys_cpu" : 75.800000,\n'
                                             '      "ctx" : 16640,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 100.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 99.999982,\n'
                                             '        "8" : 0.100000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 0.090620,\n'
                                             '        "20" : 95.460888,\n'
                                             '        "50" : 4.343226,\n'
                                             '        "100" : 0.104331,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 8,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring-xnvme: (groupid=0, '
                                             'jobs=1): err= 0: pid=11118: Wed '
                                             'Oct 26 15:28:11 2022\n'
                                             '  read: IOPS=397k, BW=194MiB/s '
                                             '(203MB/s)(1357MiB/7001msec)\n'
                                             '    slat (nsec): min=1402, '
                                             'max=29996, avg=2024.46, '
                                             'stdev=305.38\n'
                                             '    clat (usec): min=2, max=131, '
                                             'avg=17.83, stdev= 3.04\n'
                                             '     lat (usec): min=6, max=133, '
                                             'avg=19.86, stdev= 2.99\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[13248],  '
                                             '5.00th=[15168], 10.00th=[16512], '
                                             '20.00th=[17024],\n'
                                             '     | 30.00th=[17280], '
                                             '40.00th=[17536], '
                                             '50.00th=[17536], '
                                             '60.00th=[17536],\n'
                                             '     | 70.00th=[17536], '
                                             '80.00th=[18048], '
                                             '90.00th=[18304], '
                                             '95.00th=[19584],\n'
                                             '     | 99.00th=[31872], '
                                             '99.50th=[35072], '
                                             '99.90th=[50944], '
                                             '99.95th=[64768],\n'
                                             '     | 99.99th=[77312]\n'
                                             '   bw (  KiB/s): min=197185, '
                                             'max=199877, per=100.00%, '
                                             'avg=198482.57, stdev=752.01, '
                                             'samples=14\n'
                                             '   iops        : min=394370, '
                                             'max=399754, avg=396965.14, '
                                             'stdev=1504.02, samples=14\n'
                                             '  lat (usec)   : 4=0.01%, '
                                             '10=0.09%, 20=95.46%, 50=4.34%, '
                                             '100=0.10%\n'
                                             '  lat (usec)   : 250=0.01%\n'
                                             '  cpu          : usr=22.07%, '
                                             'sys=75.80%, ctx=16640, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, 2=0.0%, '
                                             '4=0.0%, 8=100.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.1%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=2778649,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=8\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=194MiB/s (203MB/s), '
                                             '194MiB/s-194MiB/s '
                                             '(203MB/s-203MB/s), io=1357MiB '
                                             '(1423MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_360.output'),
                                   'state': {'begin': 1666783801.637204,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_8_io_uring-xnvme_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 1.2167508602142334,
                                             'end': 1666783802.8539548,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_360.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_360.state')},
                       'cmd_361': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_512_8_libaio-reference_0.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_361.output'),
                                   'state': {'begin': 1666783804.3302684,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_8_libaio-reference_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 1.5104670524597168,
                                             'end': 1666783805.8407354,
                                             'err': 1,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_361.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_361.state')},
                       'cmd_362': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_362.output'),
                                   'state': {'begin': 1666783805.8427002,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="libaio-reference" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="8" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="libaio" '
                                                    '--filename="/dev/nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_8_libaio-reference_0.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.908896923065186,
                                             'end': 1666783817.7515972,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_362.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_362.state')},
                       'cmd_363': {'output': 'libaio-reference: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=libaio, iodepth=8\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666790907,\n'
                                             '  "timestamp_ms" : '
                                             '1666790907820,\n'
                                             '  "time" : "Wed Oct 26 15:28:27 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"libaio-reference",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"libaio-reference",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "8",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : "libaio",\n'
                                             '        "filename" : '
                                             '"/dev/nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '1573419008,\n'
                                             '        "io_kbytes" : 1536542,\n'
                                             '        "bw_bytes" : 224742037,\n'
                                             '        "bw" : 219474,\n'
                                             '        "iops" : 438948.293101,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 3073077,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1442,\n'
                                             '          "max" : 20178,\n'
                                             '          "mean" : 1602.113093,\n'
                                             '          "stddev" : '
                                             '122.876606,\n'
                                             '          "N" : 3073077\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 4669,\n'
                                             '          "max" : 139171,\n'
                                             '          "mean" : '
                                             '16380.314330,\n'
                                             '          "stddev" : '
                                             '3042.824860,\n'
                                             '          "N" : 3073084,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 13888,\n'
                                             '            "5.000000" : 15552,\n'
                                             '            "10.000000" : '
                                             '15680,\n'
                                             '            "20.000000" : '
                                             '15680,\n'
                                             '            "30.000000" : '
                                             '15680,\n'
                                             '            "40.000000" : '
                                             '15808,\n'
                                             '            "50.000000" : '
                                             '15808,\n'
                                             '            "60.000000" : '
                                             '15808,\n'
                                             '            "70.000000" : '
                                             '15936,\n'
                                             '            "80.000000" : '
                                             '16320,\n'
                                             '            "90.000000" : '
                                             '16512,\n'
                                             '            "95.000000" : '
                                             '17536,\n'
                                             '            "99.000000" : '
                                             '31616,\n'
                                             '            "99.500000" : '
                                             '35584,\n'
                                             '            "99.900000" : '
                                             '51456,\n'
                                             '            "99.950000" : '
                                             '66048,\n'
                                             '            "99.990000" : 78336\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 6172,\n'
                                             '          "max" : 140724,\n'
                                             '          "mean" : '
                                             '17982.427350,\n'
                                             '          "stddev" : '
                                             '3044.762520,\n'
                                             '          "N" : 3073084\n'
                                             '        },\n'
                                             '        "bw_min" : 218423,\n'
                                             '        "bw_max" : 220928,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '219536.928571,\n'
                                             '        "bw_dev" : 658.497411,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 436846,\n'
                                             '        "iops_max" : 441857,\n'
                                             '        "iops_mean" : '
                                             '439073.928571,\n'
                                             '        "iops_stddev" : '
                                             '1317.157339,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 30.700000,\n'
                                             '      "sys_cpu" : 67.314286,\n'
                                             '      "ctx" : 18155,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 100.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 99.999967,\n'
                                             '        "8" : 0.100000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.426283,\n'
                                             '        "20" : 95.925061,\n'
                                             '        "50" : 3.544721,\n'
                                             '        "100" : 0.103024,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 8,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ],\n'
                                             '  "disk_util" : [\n'
                                             '    {\n'
                                             '      "name" : "nvme0n1",\n'
                                             '      "read_ios" : 4442260,\n'
                                             '      "write_ios" : 0,\n'
                                             '      "read_merges" : 0,\n'
                                             '      "write_merges" : 0,\n'
                                             '      "read_ticks" : 25258,\n'
                                             '      "write_ticks" : 0,\n'
                                             '      "in_queue" : 25258,\n'
                                             '      "util" : 99.069123\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'libaio-reference: (groupid=0, '
                                             'jobs=1): err= 0: pid=11167: Wed '
                                             'Oct 26 15:28:27 2022\n'
                                             '  read: IOPS=439k, BW=214MiB/s '
                                             '(225MB/s)(1501MiB/7001msec)\n'
                                             '    slat (nsec): min=1442, '
                                             'max=20178, avg=1602.11, '
                                             'stdev=122.88\n'
                                             '    clat (usec): min=4, max=139, '
                                             'avg=16.38, stdev= 3.04\n'
                                             '     lat (usec): min=6, max=140, '
                                             'avg=17.98, stdev= 3.04\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[13888],  '
                                             '5.00th=[15552], 10.00th=[15680], '
                                             '20.00th=[15680],\n'
                                             '     | 30.00th=[15680], '
                                             '40.00th=[15808], '
                                             '50.00th=[15808], '
                                             '60.00th=[15808],\n'
                                             '     | 70.00th=[15936], '
                                             '80.00th=[16320], '
                                             '90.00th=[16512], '
                                             '95.00th=[17536],\n'
                                             '     | 99.00th=[31616], '
                                             '99.50th=[35584], '
                                             '99.90th=[51456], '
                                             '99.95th=[66048],\n'
                                             '     | 99.99th=[78336]\n'
                                             '   bw (  KiB/s): min=218423, '
                                             'max=220928, per=100.00%, '
                                             'avg=219536.93, stdev=658.50, '
                                             'samples=14\n'
                                             '   iops        : min=436846, '
                                             'max=441857, avg=439073.93, '
                                             'stdev=1317.16, samples=14\n'
                                             '  lat (usec)   : 10=0.43%, '
                                             '20=95.93%, 50=3.54%, 100=0.10%, '
                                             '250=0.01%\n'
                                             '  cpu          : usr=30.70%, '
                                             'sys=67.31%, ctx=18155, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, 2=0.0%, '
                                             '4=0.0%, 8=100.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.1%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=3073077,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=8\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=214MiB/s (225MB/s), '
                                             '214MiB/s-214MiB/s '
                                             '(225MB/s-225MB/s), io=1501MiB '
                                             '(1573MB), run=7001-7001msec\n'
                                             '\n'
                                             'Disk stats (read/write):\n'
                                             '  nvme0n1: ios=4442260/0, '
                                             'merge=0/0, ticks=25258/0, '
                                             'in_queue=25258, util=99.07%\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_363.output'),
                                   'state': {'begin': 1666783817.7535267,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_8_libaio-reference_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 1.55983567237854,
                                             'end': 1666783819.3133624,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_363.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_363.state')},
                       'cmd_364': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_512_8_libaio-reference_1.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_364.output'),
                                   'state': {'begin': 1666783820.844429,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_8_libaio-reference_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 1.3285760879516602,
                                             'end': 1666783822.173005,
                                             'err': 1,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_364.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_364.state')},
                       'cmd_365': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_365.output'),
                                   'state': {'begin': 1666783822.1749372,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="libaio-reference" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="8" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="libaio" '
                                                    '--filename="/dev/nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_8_libaio-reference_1.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.932475805282593,
                                             'end': 1666783834.107413,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_365.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_365.state')},
                       'cmd_366': {'output': 'libaio-reference: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=libaio, iodepth=8\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666790924,\n'
                                             '  "timestamp_ms" : '
                                             '1666790924175,\n'
                                             '  "time" : "Wed Oct 26 15:28:44 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"libaio-reference",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"libaio-reference",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "8",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : "libaio",\n'
                                             '        "filename" : '
                                             '"/dev/nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '1567867904,\n'
                                             '        "io_kbytes" : 1531121,\n'
                                             '        "bw_bytes" : 223949136,\n'
                                             '        "bw" : 218700,\n'
                                             '        "iops" : 437399.514355,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 3062234,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1442,\n'
                                             '          "max" : 80601,\n'
                                             '          "mean" : 1606.787758,\n'
                                             '          "stddev" : '
                                             '104.510126,\n'
                                             '          "N" : 3062235\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 4960,\n'
                                             '          "max" : 135585,\n'
                                             '          "mean" : '
                                             '16440.666159,\n'
                                             '          "stddev" : '
                                             '2924.034150,\n'
                                             '          "N" : 3062241,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 13760,\n'
                                             '            "5.000000" : 15680,\n'
                                             '            "10.000000" : '
                                             '15680,\n'
                                             '            "20.000000" : '
                                             '15808,\n'
                                             '            "30.000000" : '
                                             '15808,\n'
                                             '            "40.000000" : '
                                             '15808,\n'
                                             '            "50.000000" : '
                                             '15936,\n'
                                             '            "60.000000" : '
                                             '15936,\n'
                                             '            "70.000000" : '
                                             '16064,\n'
                                             '            "80.000000" : '
                                             '16512,\n'
                                             '            "90.000000" : '
                                             '16512,\n'
                                             '            "95.000000" : '
                                             '17536,\n'
                                             '            "99.000000" : '
                                             '31104,\n'
                                             '            "99.500000" : '
                                             '35072,\n'
                                             '            "99.900000" : '
                                             '48896,\n'
                                             '            "99.950000" : '
                                             '63744,\n'
                                             '            "99.990000" : 78336\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 6483,\n'
                                             '          "max" : 137268,\n'
                                             '          "mean" : '
                                             '18047.453991,\n'
                                             '          "stddev" : '
                                             '2925.765710,\n'
                                             '          "N" : 3062241\n'
                                             '        },\n'
                                             '        "bw_min" : 218165,\n'
                                             '        "bw_max" : 219778,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '218823.571429,\n'
                                             '        "bw_dev" : 414.416868,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 436330,\n'
                                             '        "iops_max" : 439557,\n'
                                             '        "iops_mean" : '
                                             '437647.285714,\n'
                                             '        "iops_stddev" : '
                                             '829.026624,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 31.600000,\n'
                                             '      "sys_cpu" : 66.528571,\n'
                                             '      "ctx" : 17763,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 100.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 99.999967,\n'
                                             '        "8" : 0.100000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.421882,\n'
                                             '        "20" : 96.022054,\n'
                                             '        "50" : 3.461231,\n'
                                             '        "100" : 0.094180,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 8,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ],\n'
                                             '  "disk_util" : [\n'
                                             '    {\n'
                                             '      "name" : "nvme0n1",\n'
                                             '      "read_ios" : 4419891,\n'
                                             '      "write_ios" : 0,\n'
                                             '      "read_merges" : 0,\n'
                                             '      "write_merges" : 0,\n'
                                             '      "read_ticks" : 25004,\n'
                                             '      "write_ticks" : 0,\n'
                                             '      "in_queue" : 25004,\n'
                                             '      "util" : 99.088748\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'libaio-reference: (groupid=0, '
                                             'jobs=1): err= 0: pid=11213: Wed '
                                             'Oct 26 15:28:44 2022\n'
                                             '  read: IOPS=437k, BW=214MiB/s '
                                             '(224MB/s)(1495MiB/7001msec)\n'
                                             '    slat (nsec): min=1442, '
                                             'max=80601, avg=1606.79, '
                                             'stdev=104.51\n'
                                             '    clat (usec): min=4, max=135, '
                                             'avg=16.44, stdev= 2.92\n'
                                             '     lat (usec): min=6, max=137, '
                                             'avg=18.05, stdev= 2.93\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[13760],  '
                                             '5.00th=[15680], 10.00th=[15680], '
                                             '20.00th=[15808],\n'
                                             '     | 30.00th=[15808], '
                                             '40.00th=[15808], '
                                             '50.00th=[15936], '
                                             '60.00th=[15936],\n'
                                             '     | 70.00th=[16064], '
                                             '80.00th=[16512], '
                                             '90.00th=[16512], '
                                             '95.00th=[17536],\n'
                                             '     | 99.00th=[31104], '
                                             '99.50th=[35072], '
                                             '99.90th=[48896], '
                                             '99.95th=[63744],\n'
                                             '     | 99.99th=[78336]\n'
                                             '   bw (  KiB/s): min=218165, '
                                             'max=219778, per=100.00%, '
                                             'avg=218823.57, stdev=414.42, '
                                             'samples=14\n'
                                             '   iops        : min=436330, '
                                             'max=439557, avg=437647.29, '
                                             'stdev=829.03, samples=14\n'
                                             '  lat (usec)   : 10=0.42%, '
                                             '20=96.02%, 50=3.46%, 100=0.09%, '
                                             '250=0.01%\n'
                                             '  cpu          : usr=31.60%, '
                                             'sys=66.53%, ctx=17763, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, 2=0.0%, '
                                             '4=0.0%, 8=100.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.1%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=3062234,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=8\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=214MiB/s (224MB/s), '
                                             '214MiB/s-214MiB/s '
                                             '(224MB/s-224MB/s), io=1495MiB '
                                             '(1568MB), run=7001-7001msec\n'
                                             '\n'
                                             'Disk stats (read/write):\n'
                                             '  nvme0n1: ios=4419891/0, '
                                             'merge=0/0, ticks=25004/0, '
                                             'in_queue=25004, util=99.09%\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_366.output'),
                                   'state': {'begin': 1666783834.1101973,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_8_libaio-reference_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.9997060298919678,
                                             'end': 1666783835.1099033,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_366.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_366.state')},
                       'cmd_367': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_512_8_libaio-reference_2.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_367.output'),
                                   'state': {'begin': 1666783836.7236187,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_8_libaio-reference_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 1.7480182647705078,
                                             'end': 1666783838.471637,
                                             'err': 1,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_367.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_367.state')},
                       'cmd_368': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_368.output'),
                                   'state': {'begin': 1666783838.4743297,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="libaio-reference" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="8" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="libaio" '
                                                    '--filename="/dev/nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_8_libaio-reference_2.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 13.069213151931763,
                                             'end': 1666783851.5435429,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_368.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_368.state')},
                       'cmd_369': {'output': 'libaio-reference: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=libaio, iodepth=8\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666790941,\n'
                                             '  "timestamp_ms" : '
                                             '1666790941607,\n'
                                             '  "time" : "Wed Oct 26 15:29:01 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"libaio-reference",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"libaio-reference",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "8",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : "libaio",\n'
                                             '        "filename" : '
                                             '"/dev/nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '1575713792,\n'
                                             '        "io_kbytes" : 1538783,\n'
                                             '        "bw_bytes" : 225069817,\n'
                                             '        "bw" : 219794,\n'
                                             '        "iops" : 439588.344522,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 3077558,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1442,\n'
                                             '          "max" : 20608,\n'
                                             '          "mean" : 1603.968562,\n'
                                             '          "stddev" : '
                                             '112.982422,\n'
                                             '          "N" : 3077559\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 4729,\n'
                                             '          "max" : 131707,\n'
                                             '          "mean" : '
                                             '16357.507495,\n'
                                             '          "stddev" : '
                                             '3033.941500,\n'
                                             '          "N" : 3077565,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 13632,\n'
                                             '            "5.000000" : 15552,\n'
                                             '            "10.000000" : '
                                             '15552,\n'
                                             '            "20.000000" : '
                                             '15680,\n'
                                             '            "30.000000" : '
                                             '15680,\n'
                                             '            "40.000000" : '
                                             '15808,\n'
                                             '            "50.000000" : '
                                             '15808,\n'
                                             '            "60.000000" : '
                                             '15808,\n'
                                             '            "70.000000" : '
                                             '15936,\n'
                                             '            "80.000000" : '
                                             '16320,\n'
                                             '            "90.000000" : '
                                             '16512,\n'
                                             '            "95.000000" : '
                                             '17536,\n'
                                             '            "99.000000" : '
                                             '31872,\n'
                                             '            "99.500000" : '
                                             '36096,\n'
                                             '            "99.900000" : '
                                             '49408,\n'
                                             '            "99.950000" : '
                                             '64256,\n'
                                             '            "99.990000" : 77312\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 6242,\n'
                                             '          "max" : 133490,\n'
                                             '          "mean" : '
                                             '17961.475945,\n'
                                             '          "stddev" : '
                                             '3035.114182,\n'
                                             '          "N" : 3077565\n'
                                             '        },\n'
                                             '        "bw_min" : 219076,\n'
                                             '        "bw_max" : 222226,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '219857.071429,\n'
                                             '        "bw_dev" : 825.530735,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 438152,\n'
                                             '        "iops_max" : 444452,\n'
                                             '        "iops_mean" : '
                                             '439714.285714,\n'
                                             '        "iops_stddev" : '
                                             '1651.104219,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 30.700000,\n'
                                             '      "sys_cpu" : 67.271429,\n'
                                             '      "ctx" : 18039,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 100.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 99.999968,\n'
                                             '        "8" : 0.100000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.440284,\n'
                                             '        "20" : 95.900321,\n'
                                             '        "50" : 3.560713,\n'
                                             '        "100" : 0.098650,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 8,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ],\n'
                                             '  "disk_util" : [\n'
                                             '    {\n'
                                             '      "name" : "nvme0n1",\n'
                                             '      "read_ios" : 4443543,\n'
                                             '      "write_ios" : 0,\n'
                                             '      "read_merges" : 0,\n'
                                             '      "write_merges" : 0,\n'
                                             '      "read_ticks" : 25340,\n'
                                             '      "write_ticks" : 0,\n'
                                             '      "in_queue" : 25340,\n'
                                             '      "util" : 99.069123\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'libaio-reference: (groupid=0, '
                                             'jobs=1): err= 0: pid=11259: Wed '
                                             'Oct 26 15:29:01 2022\n'
                                             '  read: IOPS=440k, BW=215MiB/s '
                                             '(225MB/s)(1503MiB/7001msec)\n'
                                             '    slat (nsec): min=1442, '
                                             'max=20608, avg=1603.97, '
                                             'stdev=112.98\n'
                                             '    clat (usec): min=4, max=131, '
                                             'avg=16.36, stdev= 3.03\n'
                                             '     lat (usec): min=6, max=133, '
                                             'avg=17.96, stdev= 3.04\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[13632],  '
                                             '5.00th=[15552], 10.00th=[15552], '
                                             '20.00th=[15680],\n'
                                             '     | 30.00th=[15680], '
                                             '40.00th=[15808], '
                                             '50.00th=[15808], '
                                             '60.00th=[15808],\n'
                                             '     | 70.00th=[15936], '
                                             '80.00th=[16320], '
                                             '90.00th=[16512], '
                                             '95.00th=[17536],\n'
                                             '     | 99.00th=[31872], '
                                             '99.50th=[36096], '
                                             '99.90th=[49408], '
                                             '99.95th=[64256],\n'
                                             '     | 99.99th=[77312]\n'
                                             '   bw (  KiB/s): min=219076, '
                                             'max=222226, per=100.00%, '
                                             'avg=219857.07, stdev=825.53, '
                                             'samples=14\n'
                                             '   iops        : min=438152, '
                                             'max=444452, avg=439714.29, '
                                             'stdev=1651.10, samples=14\n'
                                             '  lat (usec)   : 10=0.44%, '
                                             '20=95.90%, 50=3.56%, 100=0.10%, '
                                             '250=0.01%\n'
                                             '  cpu          : usr=30.70%, '
                                             'sys=67.27%, ctx=18039, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, 2=0.0%, '
                                             '4=0.0%, 8=100.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.1%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=3077558,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=8\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=215MiB/s (225MB/s), '
                                             '215MiB/s-215MiB/s '
                                             '(225MB/s-225MB/s), io=1503MiB '
                                             '(1576MB), run=7001-7001msec\n'
                                             '\n'
                                             'Disk stats (read/write):\n'
                                             '  nvme0n1: ios=4443543/0, '
                                             'merge=0/0, ticks=25340/0, '
                                             'in_queue=25340, util=99.07%\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_369.output'),
                                   'state': {'begin': 1666783851.5465963,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_8_libaio-reference_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 2.8698582649230957,
                                             'end': 1666783854.4164546,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_369.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_369.state')},
                       'cmd_37': {'output': 'rm: cannot remove '
                                            "'/tmp/fio-output_512_1_io_uring-bdev_uring_0.txt': "
                                            'No such file or directory\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_37.output'),
                                  'state': {'begin': 1666782217.7051482,
                                            'cmd': 'rm '
                                                   '/tmp/fio-output_512_1_io_uring-bdev_uring_0.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.85337233543396,
                                            'end': 1666782218.5585206,
                                            'err': 1,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_37.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_37.state')},
                       'cmd_370': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_512_8_libaio-bdev_aio_0.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_370.output'),
                                   'state': {'begin': 1666783857.8533573,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_8_libaio-bdev_aio_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.9677407741546631,
                                             'end': 1666783858.821098,
                                             'err': 1,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_370.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_370.state')},
                       'cmd_371': {'output': 'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_371.output'),
                                   'state': {'begin': 1666783858.8232763,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="libaio-bdev_aio" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="8" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="spdk_bdev" '
                                                    '--spdk_conf="/tmp/spdk_bdev_aio_foo.conf" '
                                                    '--filename="Nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_8_libaio-bdev_aio_0.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.803528547286987,
                                             'end': 1666783870.6268048,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_371.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_371.state')},
                       'cmd_372': {'output': 'libaio-bdev_aio: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=spdk_bdev, iodepth=8\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666790960,\n'
                                             '  "timestamp_ms" : '
                                             '1666790960594,\n'
                                             '  "time" : "Wed Oct 26 15:29:20 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"libaio-bdev_aio",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"libaio-bdev_aio",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "8",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"spdk_bdev",\n'
                                             '        "spdk_conf" : '
                                             '"/tmp/spdk_bdev_aio_foo.conf",\n'
                                             '        "filename" : "Nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '1661046784,\n'
                                             '        "io_kbytes" : 1622116,\n'
                                             '        "bw_bytes" : 237258503,\n'
                                             '        "bw" : 231697,\n'
                                             '        "iops" : 463394.372233,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 3244224,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1633,\n'
                                             '          "max" : 22713,\n'
                                             '          "mean" : 1818.265835,\n'
                                             '          "stddev" : '
                                             '123.789028,\n'
                                             '          "N" : 3244225\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 361,\n'
                                             '          "max" : 109205,\n'
                                             '          "mean" : '
                                             '10578.592328,\n'
                                             '          "stddev" : '
                                             '4535.960378,\n'
                                             '          "N" : 3244224,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 4192,\n'
                                             '            "5.000000" : 4320,\n'
                                             '            "10.000000" : 6112,\n'
                                             '            "20.000000" : 6304,\n'
                                             '            "30.000000" : 8160,\n'
                                             '            "40.000000" : 8384,\n'
                                             '            "50.000000" : '
                                             '10432,\n'
                                             '            "60.000000" : '
                                             '11200,\n'
                                             '            "70.000000" : '
                                             '12864,\n'
                                             '            "80.000000" : '
                                             '14528,\n'
                                             '            "90.000000" : '
                                             '14784,\n'
                                             '            "95.000000" : '
                                             '15296,\n'
                                             '            "99.000000" : '
                                             '26496,\n'
                                             '            "99.500000" : '
                                             '28288,\n'
                                             '            "99.900000" : '
                                             '43776,\n'
                                             '            "99.950000" : '
                                             '60160,\n'
                                             '            "99.990000" : 72192\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 5740,\n'
                                             '          "max" : 111099,\n'
                                             '          "mean" : '
                                             '12396.858125,\n'
                                             '          "stddev" : '
                                             '4531.767359,\n'
                                             '          "N" : 3244224\n'
                                             '        },\n'
                                             '        "bw_min" : 230450,\n'
                                             '        "bw_max" : 233465,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '231740.142857,\n'
                                             '        "bw_dev" : 820.889841,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 460900,\n'
                                             '        "iops_max" : 466930,\n'
                                             '        "iops_mean" : '
                                             '463480.285714,\n'
                                             '        "iops_stddev" : '
                                             '1641.459736,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 36.971429,\n'
                                             '      "sys_cpu" : 63.014286,\n'
                                             '      "ctx" : 5,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.100000,\n'
                                             '        "2" : 8.133347,\n'
                                             '        "4" : 72.486425,\n'
                                             '        "8" : 19.379550,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 85.202722,\n'
                                             '        "8" : 14.797278,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.010000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 43.187308,\n'
                                             '        "20" : 54.000433,\n'
                                             '        "50" : 2.732826,\n'
                                             '        "100" : 0.077307,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 8,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'libaio-bdev_aio: (groupid=0, '
                                             'jobs=1): err= 0: pid=11316: Wed '
                                             'Oct 26 15:29:20 2022\n'
                                             '  read: IOPS=463k, BW=226MiB/s '
                                             '(237MB/s)(1584MiB/7001msec)\n'
                                             '    slat (nsec): min=1633, '
                                             'max=22713, avg=1818.27, '
                                             'stdev=123.79\n'
                                             '    clat (nsec): min=361, '
                                             'max=109205, avg=10578.59, '
                                             'stdev=4535.96\n'
                                             '     lat (usec): min=5, max=111, '
                                             'avg=12.40, stdev= 4.53\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 4192],  '
                                             '5.00th=[ 4320], 10.00th=[ 6112], '
                                             '20.00th=[ 6304],\n'
                                             '     | 30.00th=[ 8160], '
                                             '40.00th=[ 8384], '
                                             '50.00th=[10432], '
                                             '60.00th=[11200],\n'
                                             '     | 70.00th=[12864], '
                                             '80.00th=[14528], '
                                             '90.00th=[14784], '
                                             '95.00th=[15296],\n'
                                             '     | 99.00th=[26496], '
                                             '99.50th=[28288], '
                                             '99.90th=[43776], '
                                             '99.95th=[60160],\n'
                                             '     | 99.99th=[72192]\n'
                                             '   bw (  KiB/s): min=230450, '
                                             'max=233465, per=100.00%, '
                                             'avg=231740.14, stdev=820.89, '
                                             'samples=14\n'
                                             '   iops        : min=460900, '
                                             'max=466930, avg=463480.29, '
                                             'stdev=1641.46, samples=14\n'
                                             '  lat (nsec)   : 500=0.01%\n'
                                             '  lat (usec)   : 4=0.01%, '
                                             '10=43.19%, 20=54.00%, 50=2.73%, '
                                             '100=0.08%\n'
                                             '  lat (usec)   : 250=0.01%\n'
                                             '  cpu          : usr=36.97%, '
                                             'sys=63.01%, ctx=5, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.1%, 2=8.1%, '
                                             '4=72.5%, 8=19.4%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=85.2%, 8=14.8%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=3244224,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=8\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=226MiB/s (237MB/s), '
                                             '226MiB/s-226MiB/s '
                                             '(237MB/s-237MB/s), io=1584MiB '
                                             '(1661MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_372.output'),
                                   'state': {'begin': 1666783870.6286519,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_8_libaio-bdev_aio_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 1.4652798175811768,
                                             'end': 1666783872.0939317,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_372.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_372.state')},
                       'cmd_373': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_512_8_libaio-bdev_aio_1.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_373.output'),
                                   'state': {'begin': 1666783874.6259732,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_8_libaio-bdev_aio_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 1.0284993648529053,
                                             'end': 1666783875.6544726,
                                             'err': 1,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_373.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_373.state')},
                       'cmd_374': {'output': 'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_374.output'),
                                   'state': {'begin': 1666783875.6564553,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="libaio-bdev_aio" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="8" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="spdk_bdev" '
                                                    '--spdk_conf="/tmp/spdk_bdev_aio_foo.conf" '
                                                    '--filename="Nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_8_libaio-bdev_aio_1.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.919304609298706,
                                             'end': 1666783887.57576,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_374.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_374.state')},
                       'cmd_375': {'output': 'libaio-bdev_aio: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=spdk_bdev, iodepth=8\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666790977,\n'
                                             '  "timestamp_ms" : '
                                             '1666790977538,\n'
                                             '  "time" : "Wed Oct 26 15:29:37 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"libaio-bdev_aio",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"libaio-bdev_aio",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "8",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"spdk_bdev",\n'
                                             '        "spdk_conf" : '
                                             '"/tmp/spdk_bdev_aio_foo.conf",\n'
                                             '        "filename" : "Nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '1666486272,\n'
                                             '        "io_kbytes" : 1627428,\n'
                                             '        "bw_bytes" : 238035462,\n'
                                             '        "bw" : 232456,\n'
                                             '        "iops" : 464912.012570,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 3254849,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1623,\n'
                                             '          "max" : 23605,\n'
                                             '          "mean" : 1803.010349,\n'
                                             '          "stddev" : '
                                             '126.458209,\n'
                                             '          "N" : 3254850\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 370,\n'
                                             '          "max" : 125115,\n'
                                             '          "mean" : '
                                             '10935.840299,\n'
                                             '          "stddev" : '
                                             '4372.129142,\n'
                                             '          "N" : 3254856,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 4256,\n'
                                             '            "5.000000" : 6112,\n'
                                             '            "10.000000" : 6176,\n'
                                             '            "20.000000" : 6560,\n'
                                             '            "30.000000" : 8160,\n'
                                             '            "40.000000" : '
                                             '10048,\n'
                                             '            "50.000000" : '
                                             '10688,\n'
                                             '            "60.000000" : '
                                             '12480,\n'
                                             '            "70.000000" : '
                                             '12736,\n'
                                             '            "80.000000" : '
                                             '14400,\n'
                                             '            "90.000000" : '
                                             '14656,\n'
                                             '            "95.000000" : '
                                             '15168,\n'
                                             '            "99.000000" : '
                                             '27008,\n'
                                             '            "99.500000" : '
                                             '28800,\n'
                                             '            "99.900000" : '
                                             '43264,\n'
                                             '            "99.950000" : '
                                             '60672,\n'
                                             '            "99.990000" : 72192\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 5661,\n'
                                             '          "max" : 126928,\n'
                                             '          "mean" : '
                                             '12738.850612,\n'
                                             '          "stddev" : '
                                             '4363.905228,\n'
                                             '          "N" : 3254856\n'
                                             '        },\n'
                                             '        "bw_min" : 229362,\n'
                                             '        "bw_max" : 234011,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '232498.785714,\n'
                                             '        "bw_dev" : 1354.748752,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 458724,\n'
                                             '        "iops_max" : 468022,\n'
                                             '        "iops_mean" : '
                                             '464997.642857,\n'
                                             '        "iops_stddev" : '
                                             '2709.526147,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 41.585714,\n'
                                             '      "sys_cpu" : 58.385714,\n'
                                             '      "ctx" : 7,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.100000,\n'
                                             '        "2" : 1.864664,\n'
                                             '        "4" : 77.562246,\n'
                                             '        "8" : 20.571922,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 84.411387,\n'
                                             '        "8" : 15.588613,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.010000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 39.686818,\n'
                                             '        "20" : 57.371356,\n'
                                             '        "50" : 2.863297,\n'
                                             '        "100" : 0.075702,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 8,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'libaio-bdev_aio: (groupid=0, '
                                             'jobs=1): err= 0: pid=11373: Wed '
                                             'Oct 26 15:29:37 2022\n'
                                             '  read: IOPS=465k, BW=227MiB/s '
                                             '(238MB/s)(1589MiB/7001msec)\n'
                                             '    slat (nsec): min=1623, '
                                             'max=23605, avg=1803.01, '
                                             'stdev=126.46\n'
                                             '    clat (nsec): min=370, '
                                             'max=125115, avg=10935.84, '
                                             'stdev=4372.13\n'
                                             '     lat (usec): min=5, max=126, '
                                             'avg=12.74, stdev= 4.36\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 4256],  '
                                             '5.00th=[ 6112], 10.00th=[ 6176], '
                                             '20.00th=[ 6560],\n'
                                             '     | 30.00th=[ 8160], '
                                             '40.00th=[10048], '
                                             '50.00th=[10688], '
                                             '60.00th=[12480],\n'
                                             '     | 70.00th=[12736], '
                                             '80.00th=[14400], '
                                             '90.00th=[14656], '
                                             '95.00th=[15168],\n'
                                             '     | 99.00th=[27008], '
                                             '99.50th=[28800], '
                                             '99.90th=[43264], '
                                             '99.95th=[60672],\n'
                                             '     | 99.99th=[72192]\n'
                                             '   bw (  KiB/s): min=229362, '
                                             'max=234011, per=100.00%, '
                                             'avg=232498.79, stdev=1354.75, '
                                             'samples=14\n'
                                             '   iops        : min=458724, '
                                             'max=468022, avg=464997.64, '
                                             'stdev=2709.53, samples=14\n'
                                             '  lat (nsec)   : 500=0.01%\n'
                                             '  lat (usec)   : 4=0.01%, '
                                             '10=39.69%, 20=57.37%, 50=2.86%, '
                                             '100=0.08%\n'
                                             '  lat (usec)   : 250=0.01%\n'
                                             '  cpu          : usr=41.59%, '
                                             'sys=58.39%, ctx=7, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.1%, 2=1.9%, '
                                             '4=77.6%, 8=20.6%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=84.4%, 8=15.6%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=3254849,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=8\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=227MiB/s (238MB/s), '
                                             '227MiB/s-227MiB/s '
                                             '(238MB/s-238MB/s), io=1589MiB '
                                             '(1666MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_375.output'),
                                   'state': {'begin': 1666783887.5784094,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_8_libaio-bdev_aio_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8194758892059326,
                                             'end': 1666783888.3978853,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_375.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_375.state')},
                       'cmd_376': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_512_8_libaio-bdev_aio_2.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_376.output'),
                                   'state': {'begin': 1666783890.4773853,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_8_libaio-bdev_aio_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8124725818634033,
                                             'end': 1666783891.2898579,
                                             'err': 1,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_376.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_376.state')},
                       'cmd_377': {'output': 'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_377.output'),
                                   'state': {'begin': 1666783891.2919939,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="libaio-bdev_aio" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="8" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="spdk_bdev" '
                                                    '--spdk_conf="/tmp/spdk_bdev_aio_foo.conf" '
                                                    '--filename="Nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_8_libaio-bdev_aio_2.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.5554780960083,
                                             'end': 1666783902.847472,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_377.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_377.state')},
                       'cmd_378': {'output': 'libaio-bdev_aio: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=spdk_bdev, iodepth=8\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666790992,\n'
                                             '  "timestamp_ms" : '
                                             '1666790992810,\n'
                                             '  "time" : "Wed Oct 26 15:29:52 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"libaio-bdev_aio",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"libaio-bdev_aio",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "8",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"spdk_bdev",\n'
                                             '        "spdk_conf" : '
                                             '"/tmp/spdk_bdev_aio_foo.conf",\n'
                                             '        "filename" : "Nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '1667806208,\n'
                                             '        "io_kbytes" : 1628717,\n'
                                             '        "bw_bytes" : 238223997,\n'
                                             '        "bw" : 232640,\n'
                                             '        "iops" : 465280.531353,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 3257429,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1623,\n'
                                             '          "max" : 23795,\n'
                                             '          "mean" : 1807.492342,\n'
                                             '          "stddev" : '
                                             '121.710490,\n'
                                             '          "N" : 3257430\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 391,\n'
                                             '          "max" : 110808,\n'
                                             '          "mean" : '
                                             '10528.263130,\n'
                                             '          "stddev" : '
                                             '4533.572349,\n'
                                             '          "N" : 3257434,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 4192,\n'
                                             '            "5.000000" : 4256,\n'
                                             '            "10.000000" : 6112,\n'
                                             '            "20.000000" : 6240,\n'
                                             '            "30.000000" : 8096,\n'
                                             '            "40.000000" : 8256,\n'
                                             '            "50.000000" : '
                                             '10304,\n'
                                             '            "60.000000" : '
                                             '11200,\n'
                                             '            "70.000000" : '
                                             '12736,\n'
                                             '            "80.000000" : '
                                             '14528,\n'
                                             '            "90.000000" : '
                                             '14784,\n'
                                             '            "95.000000" : '
                                             '15296,\n'
                                             '            "99.000000" : '
                                             '26240,\n'
                                             '            "99.500000" : '
                                             '28288,\n'
                                             '            "99.900000" : '
                                             '43264,\n'
                                             '            "99.950000" : '
                                             '60672,\n'
                                             '            "99.990000" : 72192\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 5650,\n'
                                             '          "max" : 112531,\n'
                                             '          "mean" : '
                                             '12335.755434,\n'
                                             '          "stddev" : '
                                             '4529.016440,\n'
                                             '          "N" : 3257434\n'
                                             '        },\n'
                                             '        "bw_min" : 230862,\n'
                                             '        "bw_max" : 235114,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '232693.000000,\n'
                                             '        "bw_dev" : 1547.197367,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 461716,\n'
                                             '        "iops_max" : 470228,\n'
                                             '        "iops_mean" : '
                                             '465386.000000,\n'
                                             '        "iops_stddev" : '
                                             '3095.091101,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 38.057143,\n'
                                             '      "sys_cpu" : 61.928571,\n'
                                             '      "ctx" : 4,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.100000,\n'
                                             '        "2" : 8.325922,\n'
                                             '        "4" : 72.295359,\n'
                                             '        "8" : 19.377736,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 85.258211,\n'
                                             '        "8" : 14.741789,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.010000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 43.861094,\n'
                                             '        "20" : 53.338047,\n'
                                             '        "50" : 2.724081,\n'
                                             '        "100" : 0.074261,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 8,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'libaio-bdev_aio: (groupid=0, '
                                             'jobs=1): err= 0: pid=11431: Wed '
                                             'Oct 26 15:29:52 2022\n'
                                             '  read: IOPS=465k, BW=227MiB/s '
                                             '(238MB/s)(1591MiB/7001msec)\n'
                                             '    slat (nsec): min=1623, '
                                             'max=23795, avg=1807.49, '
                                             'stdev=121.71\n'
                                             '    clat (nsec): min=391, '
                                             'max=110808, avg=10528.26, '
                                             'stdev=4533.57\n'
                                             '     lat (usec): min=5, max=112, '
                                             'avg=12.34, stdev= 4.53\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 4192],  '
                                             '5.00th=[ 4256], 10.00th=[ 6112], '
                                             '20.00th=[ 6240],\n'
                                             '     | 30.00th=[ 8096], '
                                             '40.00th=[ 8256], '
                                             '50.00th=[10304], '
                                             '60.00th=[11200],\n'
                                             '     | 70.00th=[12736], '
                                             '80.00th=[14528], '
                                             '90.00th=[14784], '
                                             '95.00th=[15296],\n'
                                             '     | 99.00th=[26240], '
                                             '99.50th=[28288], '
                                             '99.90th=[43264], '
                                             '99.95th=[60672],\n'
                                             '     | 99.99th=[72192]\n'
                                             '   bw (  KiB/s): min=230862, '
                                             'max=235114, per=100.00%, '
                                             'avg=232693.00, stdev=1547.20, '
                                             'samples=14\n'
                                             '   iops        : min=461716, '
                                             'max=470228, avg=465386.00, '
                                             'stdev=3095.09, samples=14\n'
                                             '  lat (nsec)   : 500=0.01%\n'
                                             '  lat (usec)   : 4=0.01%, '
                                             '10=43.86%, 20=53.34%, 50=2.72%, '
                                             '100=0.07%\n'
                                             '  lat (usec)   : 250=0.01%\n'
                                             '  cpu          : usr=38.06%, '
                                             'sys=61.93%, ctx=4, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.1%, 2=8.3%, '
                                             '4=72.3%, 8=19.4%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=85.3%, 8=14.7%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=3257429,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=8\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=227MiB/s (238MB/s), '
                                             '227MiB/s-227MiB/s '
                                             '(238MB/s-238MB/s), io=1591MiB '
                                             '(1668MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_378.output'),
                                   'state': {'begin': 1666783902.8541772,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_8_libaio-bdev_aio_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8841710090637207,
                                             'end': 1666783903.7383482,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_378.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_378.state')},
                       'cmd_379': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_512_8_libaio-bdev_xnvme_0.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_379.output'),
                                   'state': {'begin': 1666783905.848171,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_8_libaio-bdev_xnvme_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8557703495025635,
                                             'end': 1666783906.7039413,
                                             'err': 1,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_379.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_379.state')},
                       'cmd_38': {'output': 'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_38.output'),
                                  'state': {'begin': 1666782218.5652142,
                                            'cmd': '/opt/fio/bin/fio '
                                                   '--name="io_uring-bdev_uring" '
                                                   '--rw="randread" '
                                                   '--size="12G" --iodepth="1" '
                                                   '--bs="512" --direct="1" '
                                                   '--thread="1" '
                                                   '--time_based="1" '
                                                   '--runtime="7" '
                                                   '--ramp_time="3" '
                                                   '--norandommap="1" '
                                                   '--allow_file_create="0" '
                                                   '--ioengine="spdk_bdev" '
                                                   '--spdk_conf="/tmp/spdk_bdev_uring_foo.conf" '
                                                   '--filename="Nvme0n1" '
                                                   '--output-format="normal,json" '
                                                   '--output="/tmp/fio-output_512_1_io_uring-bdev_uring_0.txt" '
                                                   '--cpus_allowed="1"',
                                            'cwd': 'None',
                                            'elapsed': 11.326873302459717,
                                            'end': 1666782229.8920875,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_38.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_38.state')},
                       'cmd_380': {'output': 'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_380.output'),
                                   'state': {'begin': 1666783906.7059243,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="libaio-bdev_xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="8" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="spdk_bdev" '
                                                    '--spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" '
                                                    '--filename="Nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_8_libaio-bdev_xnvme_0.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.521454334259033,
                                             'end': 1666783918.2273786,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_380.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_380.state')},
                       'cmd_381': {'output': 'libaio-bdev_xnvme: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=spdk_bdev, iodepth=8\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666791008,\n'
                                             '  "timestamp_ms" : '
                                             '1666791008196,\n'
                                             '  "time" : "Wed Oct 26 15:30:08 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"libaio-bdev_xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"libaio-bdev_xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "8",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"spdk_bdev",\n'
                                             '        "spdk_conf" : '
                                             '"/tmp/spdk_bdev_xnvme_libaio.conf",\n'
                                             '        "filename" : "Nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '1567925248,\n'
                                             '        "io_kbytes" : 1531177,\n'
                                             '        "bw_bytes" : 223957327,\n'
                                             '        "bw" : 218708,\n'
                                             '        "iops" : 437415.940580,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 3062349,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1663,\n'
                                             '          "max" : 34314,\n'
                                             '          "mean" : 1858.695535,\n'
                                             '          "stddev" : '
                                             '122.078812,\n'
                                             '          "N" : 3062350\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 831,\n'
                                             '          "max" : 122730,\n'
                                             '          "mean" : '
                                             '11448.851285,\n'
                                             '          "stddev" : '
                                             '4591.776289,\n'
                                             '          "N" : 3062354,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 4704,\n'
                                             '            "5.000000" : 4896,\n'
                                             '            "10.000000" : 6688,\n'
                                             '            "20.000000" : 6880,\n'
                                             '            "30.000000" : 8768,\n'
                                             '            "40.000000" : 9152,\n'
                                             '            "50.000000" : '
                                             '11584,\n'
                                             '            "60.000000" : '
                                             '12352,\n'
                                             '            "70.000000" : '
                                             '13888,\n'
                                             '            "80.000000" : '
                                             '15680,\n'
                                             '            "90.000000" : '
                                             '16064,\n'
                                             '            "95.000000" : '
                                             '16768,\n'
                                             '            "99.000000" : '
                                             '26240,\n'
                                             '            "99.500000" : '
                                             '29056,\n'
                                             '            "99.900000" : '
                                             '44800,\n'
                                             '            "99.950000" : '
                                             '59648,\n'
                                             '            "99.990000" : 72192\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 5661,\n'
                                             '          "max" : 124534,\n'
                                             '          "mean" : '
                                             '13307.546791,\n'
                                             '          "stddev" : '
                                             '4584.897019,\n'
                                             '          "N" : 3062354\n'
                                             '        },\n'
                                             '        "bw_min" : 218159,\n'
                                             '        "bw_max" : 219551,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '218814.142857,\n'
                                             '        "bw_dev" : 384.929535,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 436318,\n'
                                             '        "iops_max" : 439102,\n'
                                             '        "iops_mean" : '
                                             '437628.428571,\n'
                                             '        "iops_stddev" : '
                                             '769.682623,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 39.185714,\n'
                                             '      "sys_cpu" : 60.785714,\n'
                                             '      "ctx" : 10,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.100000,\n'
                                             '        "2" : 7.306744,\n'
                                             '        "4" : 73.195772,\n'
                                             '        "8" : 19.496700,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 85.097695,\n'
                                             '        "8" : 14.902305,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.010000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 42.621955,\n'
                                             '        "20" : 54.869220,\n'
                                             '        "50" : 2.423107,\n'
                                             '        "100" : 0.082584,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 8,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'libaio-bdev_xnvme: (groupid=0, '
                                             'jobs=1): err= 0: pid=11488: Wed '
                                             'Oct 26 15:30:08 2022\n'
                                             '  read: IOPS=437k, BW=214MiB/s '
                                             '(224MB/s)(1495MiB/7001msec)\n'
                                             '    slat (nsec): min=1663, '
                                             'max=34314, avg=1858.70, '
                                             'stdev=122.08\n'
                                             '    clat (nsec): min=831, '
                                             'max=122730, avg=11448.85, '
                                             'stdev=4591.78\n'
                                             '     lat (usec): min=5, max=124, '
                                             'avg=13.31, stdev= 4.58\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 4704],  '
                                             '5.00th=[ 4896], 10.00th=[ 6688], '
                                             '20.00th=[ 6880],\n'
                                             '     | 30.00th=[ 8768], '
                                             '40.00th=[ 9152], '
                                             '50.00th=[11584], '
                                             '60.00th=[12352],\n'
                                             '     | 70.00th=[13888], '
                                             '80.00th=[15680], '
                                             '90.00th=[16064], '
                                             '95.00th=[16768],\n'
                                             '     | 99.00th=[26240], '
                                             '99.50th=[29056], '
                                             '99.90th=[44800], '
                                             '99.95th=[59648],\n'
                                             '     | 99.99th=[72192]\n'
                                             '   bw (  KiB/s): min=218159, '
                                             'max=219551, per=100.00%, '
                                             'avg=218814.14, stdev=384.93, '
                                             'samples=14\n'
                                             '   iops        : min=436318, '
                                             'max=439102, avg=437628.43, '
                                             'stdev=769.68, samples=14\n'
                                             '  lat (nsec)   : 1000=0.01%\n'
                                             '  lat (usec)   : 4=0.01%, '
                                             '10=42.62%, 20=54.87%, 50=2.42%, '
                                             '100=0.08%\n'
                                             '  lat (usec)   : 250=0.01%\n'
                                             '  cpu          : usr=39.19%, '
                                             'sys=60.79%, ctx=10, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.1%, 2=7.3%, '
                                             '4=73.2%, 8=19.5%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=85.1%, 8=14.9%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=3062349,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=8\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=214MiB/s (224MB/s), '
                                             '214MiB/s-214MiB/s '
                                             '(224MB/s-224MB/s), io=1495MiB '
                                             '(1568MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_381.output'),
                                   'state': {'begin': 1666783918.2299788,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_8_libaio-bdev_xnvme_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.828787088394165,
                                             'end': 1666783919.058766,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_381.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_381.state')},
                       'cmd_382': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_512_8_libaio-bdev_xnvme_1.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_382.output'),
                                   'state': {'begin': 1666783921.2030697,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_8_libaio-bdev_xnvme_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8114423751831055,
                                             'end': 1666783922.014512,
                                             'err': 1,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_382.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_382.state')},
                       'cmd_383': {'output': 'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_383.output'),
                                   'state': {'begin': 1666783922.0164816,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="libaio-bdev_xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="8" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="spdk_bdev" '
                                                    '--spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" '
                                                    '--filename="Nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_8_libaio-bdev_xnvme_1.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.534029483795166,
                                             'end': 1666783933.5505111,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_383.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_383.state')},
                       'cmd_384': {'output': 'libaio-bdev_xnvme: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=spdk_bdev, iodepth=8\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666791023,\n'
                                             '  "timestamp_ms" : '
                                             '1666791023513,\n'
                                             '  "time" : "Wed Oct 26 15:30:23 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"libaio-bdev_xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"libaio-bdev_xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "8",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"spdk_bdev",\n'
                                             '        "spdk_conf" : '
                                             '"/tmp/spdk_bdev_xnvme_libaio.conf",\n'
                                             '        "filename" : "Nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '1576878592,\n'
                                             '        "io_kbytes" : 1539920,\n'
                                             '        "bw_bytes" : 225236193,\n'
                                             '        "bw" : 219957,\n'
                                             '        "iops" : 439913.726610,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 3079836,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1663,\n'
                                             '          "max" : 19526,\n'
                                             '          "mean" : 1853.081294,\n'
                                             '          "stddev" : '
                                             '134.461061,\n'
                                             '          "N" : 3079837\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 812,\n'
                                             '          "max" : 111990,\n'
                                             '          "mean" : '
                                             '11416.888518,\n'
                                             '          "stddev" : '
                                             '4588.974480,\n'
                                             '          "N" : 3079841,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 4704,\n'
                                             '            "5.000000" : 4896,\n'
                                             '            "10.000000" : 6624,\n'
                                             '            "20.000000" : 6816,\n'
                                             '            "30.000000" : 8768,\n'
                                             '            "40.000000" : 9152,\n'
                                             '            "50.000000" : '
                                             '11584,\n'
                                             '            "60.000000" : '
                                             '12352,\n'
                                             '            "70.000000" : '
                                             '13888,\n'
                                             '            "80.000000" : '
                                             '15552,\n'
                                             '            "90.000000" : '
                                             '15936,\n'
                                             '            "95.000000" : '
                                             '16512,\n'
                                             '            "99.000000" : '
                                             '26752,\n'
                                             '            "99.500000" : '
                                             '29312,\n'
                                             '            "99.900000" : '
                                             '44288,\n'
                                             '            "99.950000" : '
                                             '60160,\n'
                                             '            "99.990000" : 73216\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 5660,\n'
                                             '          "max" : 113774,\n'
                                             '          "mean" : '
                                             '13269.969731,\n'
                                             '          "stddev" : '
                                             '4580.597825,\n'
                                             '          "N" : 3079841\n'
                                             '        },\n'
                                             '        "bw_min" : 219083,\n'
                                             '        "bw_max" : 221465,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '219996.928571,\n'
                                             '        "bw_dev" : 649.316737,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 438166,\n'
                                             '        "iops_max" : 442931,\n'
                                             '        "iops_mean" : '
                                             '439993.928571,\n'
                                             '        "iops_stddev" : '
                                             '1298.760028,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 37.185714,\n'
                                             '      "sys_cpu" : 62.800000,\n'
                                             '      "ctx" : 5,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.100000,\n'
                                             '        "2" : 6.747048,\n'
                                             '        "4" : 73.657980,\n'
                                             '        "8" : 19.593803,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 85.015234,\n'
                                             '        "8" : 14.984766,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.010000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 42.298421,\n'
                                             '        "20" : 55.114980,\n'
                                             '        "50" : 2.504776,\n'
                                             '        "100" : 0.078511,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 8,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'libaio-bdev_xnvme: (groupid=0, '
                                             'jobs=1): err= 0: pid=11545: Wed '
                                             'Oct 26 15:30:23 2022\n'
                                             '  read: IOPS=440k, BW=215MiB/s '
                                             '(225MB/s)(1504MiB/7001msec)\n'
                                             '    slat (nsec): min=1663, '
                                             'max=19526, avg=1853.08, '
                                             'stdev=134.46\n'
                                             '    clat (nsec): min=812, '
                                             'max=111990, avg=11416.89, '
                                             'stdev=4588.97\n'
                                             '     lat (usec): min=5, max=113, '
                                             'avg=13.27, stdev= 4.58\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 4704],  '
                                             '5.00th=[ 4896], 10.00th=[ 6624], '
                                             '20.00th=[ 6816],\n'
                                             '     | 30.00th=[ 8768], '
                                             '40.00th=[ 9152], '
                                             '50.00th=[11584], '
                                             '60.00th=[12352],\n'
                                             '     | 70.00th=[13888], '
                                             '80.00th=[15552], '
                                             '90.00th=[15936], '
                                             '95.00th=[16512],\n'
                                             '     | 99.00th=[26752], '
                                             '99.50th=[29312], '
                                             '99.90th=[44288], '
                                             '99.95th=[60160],\n'
                                             '     | 99.99th=[73216]\n'
                                             '   bw (  KiB/s): min=219083, '
                                             'max=221465, per=100.00%, '
                                             'avg=219996.93, stdev=649.32, '
                                             'samples=14\n'
                                             '   iops        : min=438166, '
                                             'max=442931, avg=439993.93, '
                                             'stdev=1298.76, samples=14\n'
                                             '  lat (nsec)   : 1000=0.01%\n'
                                             '  lat (usec)   : 4=0.01%, '
                                             '10=42.30%, 20=55.11%, 50=2.50%, '
                                             '100=0.08%\n'
                                             '  lat (usec)   : 250=0.01%\n'
                                             '  cpu          : usr=37.19%, '
                                             'sys=62.80%, ctx=5, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.1%, 2=6.7%, '
                                             '4=73.7%, 8=19.6%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=85.0%, 8=15.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=3079836,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=8\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=215MiB/s (225MB/s), '
                                             '215MiB/s-215MiB/s '
                                             '(225MB/s-225MB/s), io=1504MiB '
                                             '(1577MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_384.output'),
                                   'state': {'begin': 1666783933.554262,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_8_libaio-bdev_xnvme_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8546414375305176,
                                             'end': 1666783934.4089034,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_384.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_384.state')},
                       'cmd_385': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_512_8_libaio-bdev_xnvme_2.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_385.output'),
                                   'state': {'begin': 1666783936.581162,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_8_libaio-bdev_xnvme_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8732352256774902,
                                             'end': 1666783937.4543972,
                                             'err': 1,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_385.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_385.state')},
                       'cmd_386': {'output': 'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_386.output'),
                                   'state': {'begin': 1666783937.4561808,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="libaio-bdev_xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="8" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="spdk_bdev" '
                                                    '--spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" '
                                                    '--filename="Nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_8_libaio-bdev_xnvme_2.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.624598264694214,
                                             'end': 1666783949.080779,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_386.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_386.state')},
                       'cmd_387': {'output': 'libaio-bdev_xnvme: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=spdk_bdev, iodepth=8\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666791039,\n'
                                             '  "timestamp_ms" : '
                                             '1666791039010,\n'
                                             '  "time" : "Wed Oct 26 15:30:39 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"libaio-bdev_xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"libaio-bdev_xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "8",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"spdk_bdev",\n'
                                             '        "spdk_conf" : '
                                             '"/tmp/spdk_bdev_xnvme_libaio.conf",\n'
                                             '        "filename" : "Nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '1597813760,\n'
                                             '        "io_kbytes" : 1560365,\n'
                                             '        "bw_bytes" : 228226504,\n'
                                             '        "bw" : 222877,\n'
                                             '        "iops" : 445754.320811,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 3120726,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1653,\n'
                                             '          "max" : 42329,\n'
                                             '          "mean" : 1834.530434,\n'
                                             '          "stddev" : '
                                             '111.510378,\n'
                                             '          "N" : 3120727\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 791,\n'
                                             '          "max" : 125785,\n'
                                             '          "mean" : '
                                             '11024.351078,\n'
                                             '          "stddev" : '
                                             '4765.287623,\n'
                                             '          "N" : 3120730,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 4640,\n'
                                             '            "5.000000" : 4704,\n'
                                             '            "10.000000" : 4896,\n'
                                             '            "20.000000" : 6688,\n'
                                             '            "30.000000" : 8640,\n'
                                             '            "40.000000" : 8768,\n'
                                             '            "50.000000" : '
                                             '10688,\n'
                                             '            "60.000000" : '
                                             '11712,\n'
                                             '            "70.000000" : '
                                             '13632,\n'
                                             '            "80.000000" : '
                                             '15424,\n'
                                             '            "90.000000" : '
                                             '15808,\n'
                                             '            "95.000000" : '
                                             '16320,\n'
                                             '            "99.000000" : '
                                             '27520,\n'
                                             '            "99.500000" : '
                                             '29824,\n'
                                             '            "99.900000" : '
                                             '45312,\n'
                                             '            "99.950000" : '
                                             '61184,\n'
                                             '            "99.990000" : 74240\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 5721,\n'
                                             '          "max" : 127559,\n'
                                             '          "mean" : '
                                             '12858.881482,\n'
                                             '          "stddev" : '
                                             '4762.175625,\n'
                                             '          "N" : 3120730\n'
                                             '        },\n'
                                             '        "bw_min" : 221033,\n'
                                             '        "bw_max" : 224499,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '222992.500000,\n'
                                             '        "bw_dev" : 1021.068049,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 442066,\n'
                                             '        "iops_max" : 448998,\n'
                                             '        "iops_mean" : '
                                             '445985.000000,\n'
                                             '        "iops_stddev" : '
                                             '2042.136099,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 36.557143,\n'
                                             '      "sys_cpu" : 63.414286,\n'
                                             '      "ctx" : 13,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.100000,\n'
                                             '        "2" : 10.780793,\n'
                                             '        "4" : 70.423421,\n'
                                             '        "8" : 18.795178,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 85.618620,\n'
                                             '        "8" : 14.381380,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.010000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 44.750004,\n'
                                             '        "20" : 52.543575,\n'
                                             '        "50" : 2.621249,\n'
                                             '        "100" : 0.082577,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 8,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'libaio-bdev_xnvme: (groupid=0, '
                                             'jobs=1): err= 0: pid=11603: Wed '
                                             'Oct 26 15:30:39 2022\n'
                                             '  read: IOPS=446k, BW=218MiB/s '
                                             '(228MB/s)(1524MiB/7001msec)\n'
                                             '    slat (nsec): min=1653, '
                                             'max=42329, avg=1834.53, '
                                             'stdev=111.51\n'
                                             '    clat (nsec): min=791, '
                                             'max=125785, avg=11024.35, '
                                             'stdev=4765.29\n'
                                             '     lat (usec): min=5, max=127, '
                                             'avg=12.86, stdev= 4.76\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 4640],  '
                                             '5.00th=[ 4704], 10.00th=[ 4896], '
                                             '20.00th=[ 6688],\n'
                                             '     | 30.00th=[ 8640], '
                                             '40.00th=[ 8768], '
                                             '50.00th=[10688], '
                                             '60.00th=[11712],\n'
                                             '     | 70.00th=[13632], '
                                             '80.00th=[15424], '
                                             '90.00th=[15808], '
                                             '95.00th=[16320],\n'
                                             '     | 99.00th=[27520], '
                                             '99.50th=[29824], '
                                             '99.90th=[45312], '
                                             '99.95th=[61184],\n'
                                             '     | 99.99th=[74240]\n'
                                             '   bw (  KiB/s): min=221033, '
                                             'max=224499, per=100.00%, '
                                             'avg=222992.50, stdev=1021.07, '
                                             'samples=14\n'
                                             '   iops        : min=442066, '
                                             'max=448998, avg=445985.00, '
                                             'stdev=2042.14, samples=14\n'
                                             '  lat (nsec)   : 1000=0.01%\n'
                                             '  lat (usec)   : 4=0.01%, '
                                             '10=44.75%, 20=52.54%, 50=2.62%, '
                                             '100=0.08%\n'
                                             '  lat (usec)   : 250=0.01%\n'
                                             '  cpu          : usr=36.56%, '
                                             'sys=63.41%, ctx=13, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.1%, '
                                             '2=10.8%, 4=70.4%, 8=18.8%, '
                                             '16=0.0%, 32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=85.6%, 8=14.4%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=3120726,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=8\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=218MiB/s (228MB/s), '
                                             '218MiB/s-218MiB/s '
                                             '(228MB/s-228MB/s), io=1524MiB '
                                             '(1598MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_387.output'),
                                   'state': {'begin': 1666783949.083425,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_8_libaio-bdev_xnvme_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8751075267791748,
                                             'end': 1666783949.9585326,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_387.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_387.state')},
                       'cmd_388': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_512_8_libaio-xnvme_0.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_388.output'),
                                   'state': {'begin': 1666783951.007023,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_8_libaio-xnvme_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8368914127349854,
                                             'end': 1666783951.8439145,
                                             'err': 1,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_388.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_388.state')},
                       'cmd_389': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_389.output'),
                                   'state': {'begin': 1666783951.845953,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="libaio-xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="8" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
                                                    '--xnvme_be="linux" '
                                                    '--xnvme_async="libaio" '
                                                    '--xnvme_sync="nvme" '
                                                    '--xnvme_admin="nvme" '
                                                    '--xnvme_dev_nsid="1" '
                                                    '--filename="/dev/nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_8_libaio-xnvme_0.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.118852138519287,
                                             'end': 1666783962.9648051,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_389.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_389.state')},
                       'cmd_39': {'output': 'io_uring-bdev_uring: (g=0): '
                                            'rw=randread, bs=(R) 512B-512B, '
                                            '(W) 512B-512B, (T) 512B-512B, '
                                            'ioengine=spdk_bdev, iodepth=1\n'
                                            'fio-3.32-dirty\n'
                                            'Starting 1 thread\n'
                                            '{\n'
                                            '  "fio version" : '
                                            '"fio-3.32-dirty",\n'
                                            '  "timestamp" : 1666789319,\n'
                                            '  "timestamp_ms" : '
                                            '1666789319922,\n'
                                            '  "time" : "Wed Oct 26 15:01:59 '
                                            '2022",\n'
                                            '  "jobs" : [\n'
                                            '    {\n'
                                            '      "jobname" : '
                                            '"io_uring-bdev_uring",\n'
                                            '      "groupid" : 0,\n'
                                            '      "error" : 0,\n'
                                            '      "eta" : 0,\n'
                                            '      "elapsed" : 11,\n'
                                            '      "job options" : {\n'
                                            '        "name" : '
                                            '"io_uring-bdev_uring",\n'
                                            '        "rw" : "randread",\n'
                                            '        "size" : "12G",\n'
                                            '        "iodepth" : "1",\n'
                                            '        "bs" : "512",\n'
                                            '        "direct" : "1",\n'
                                            '        "thread" : "1",\n'
                                            '        "time_based" : "1",\n'
                                            '        "runtime" : "7",\n'
                                            '        "ramp_time" : "3",\n'
                                            '        "norandommap" : "1",\n'
                                            '        "allow_file_create" : '
                                            '"0",\n'
                                            '        "ioengine" : '
                                            '"spdk_bdev",\n'
                                            '        "spdk_conf" : '
                                            '"/tmp/spdk_bdev_uring_foo.conf",\n'
                                            '        "filename" : "Nvme0n1",\n'
                                            '        "cpus_allowed" : "1"\n'
                                            '      },\n'
                                            '      "read" : {\n'
                                            '        "io_bytes" : 484563456,\n'
                                            '        "io_kbytes" : 473206,\n'
                                            '        "bw_bytes" : 69213463,\n'
                                            '        "bw" : 67591,\n'
                                            '        "iops" : 135182.545351,\n'
                                            '        "runtime" : 7001,\n'
                                            '        "total_ios" : 946413,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 150,\n'
                                            '          "max" : 9447,\n'
                                            '          "mean" : 172.883611,\n'
                                            '          "stddev" : 23.288240,\n'
                                            '          "N" : 946413\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 6261,\n'
                                            '          "max" : 85470,\n'
                                            '          "mean" : 6992.056989,\n'
                                            '          "stddev" : '
                                            '1186.274005,\n'
                                            '          "N" : 946413,\n'
                                            '          "percentile" : {\n'
                                            '            "1.000000" : 6496,\n'
                                            '            "5.000000" : 6688,\n'
                                            '            "10.000000" : 6752,\n'
                                            '            "20.000000" : 6752,\n'
                                            '            "30.000000" : 6752,\n'
                                            '            "40.000000" : 6816,\n'
                                            '            "50.000000" : 6816,\n'
                                            '            "60.000000" : 6880,\n'
                                            '            "70.000000" : 6880,\n'
                                            '            "80.000000" : 6944,\n'
                                            '            "90.000000" : 7008,\n'
                                            '            "95.000000" : 7136,\n'
                                            '            "99.000000" : 13120,\n'
                                            '            "99.500000" : 14784,\n'
                                            '            "99.900000" : 17024,\n'
                                            '            "99.950000" : 17280,\n'
                                            '            "99.990000" : 60672\n'
                                            '          }\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 6432,\n'
                                            '          "max" : 85640,\n'
                                            '          "mean" : 7164.940600,\n'
                                            '          "stddev" : '
                                            '1186.749310,\n'
                                            '          "N" : 946413\n'
                                            '        },\n'
                                            '        "bw_min" : 67403,\n'
                                            '        "bw_max" : 67839,\n'
                                            '        "bw_agg" : 100.000000,\n'
                                            '        "bw_mean" : '
                                            '67627.785714,\n'
                                            '        "bw_dev" : 111.226850,\n'
                                            '        "bw_samples" : 14,\n'
                                            '        "iops_min" : 134806,\n'
                                            '        "iops_max" : 135679,\n'
                                            '        "iops_mean" : '
                                            '135255.642857,\n'
                                            '        "iops_stddev" : '
                                            '222.599885,\n'
                                            '        "iops_samples" : 14\n'
                                            '      },\n'
                                            '      "write" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "trim" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "sync" : {\n'
                                            '        "total_ios" : 0,\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        }\n'
                                            '      },\n'
                                            '      "job_runtime" : 7000,\n'
                                            '      "usr_cpu" : 53.014286,\n'
                                            '      "sys_cpu" : 46.957143,\n'
                                            '      "ctx" : 10,\n'
                                            '      "majf" : 0,\n'
                                            '      "minf" : 0,\n'
                                            '      "iodepth_level" : {\n'
                                            '        "1" : 100.000000,\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_submit" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_complete" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ns" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_us" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 98.381151,\n'
                                            '        "20" : 1.599090,\n'
                                            '        "50" : 0.010000,\n'
                                            '        "100" : 0.015955,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ms" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000,\n'
                                            '        "2000" : 0.000000,\n'
                                            '        ">=2000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_depth" : 1,\n'
                                            '      "latency_target" : 0,\n'
                                            '      "latency_percentile" : '
                                            '100.000000,\n'
                                            '      "latency_window" : 0\n'
                                            '    }\n'
                                            '  ]\n'
                                            '}\n'
                                            '\n'
                                            'io_uring-bdev_uring: (groupid=0, '
                                            'jobs=1): err= 0: pid=5540: Wed '
                                            'Oct 26 15:01:59 2022\n'
                                            '  read: IOPS=135k, BW=66.0MiB/s '
                                            '(69.2MB/s)(462MiB/7001msec)\n'
                                            '    slat (nsec): min=150, '
                                            'max=9447, avg=172.88, '
                                            'stdev=23.29\n'
                                            '    clat (nsec): min=6261, '
                                            'max=85470, avg=6992.06, '
                                            'stdev=1186.27\n'
                                            '     lat (nsec): min=6432, '
                                            'max=85640, avg=7164.94, '
                                            'stdev=1186.75\n'
                                            '    clat percentiles (nsec):\n'
                                            '     |  1.00th=[ 6496],  5.00th=[ '
                                            '6688], 10.00th=[ 6752], 20.00th=[ '
                                            '6752],\n'
                                            '     | 30.00th=[ 6752], 40.00th=[ '
                                            '6816], 50.00th=[ 6816], 60.00th=[ '
                                            '6880],\n'
                                            '     | 70.00th=[ 6880], 80.00th=[ '
                                            '6944], 90.00th=[ 7008], 95.00th=[ '
                                            '7136],\n'
                                            '     | 99.00th=[13120], '
                                            '99.50th=[14784], 99.90th=[17024], '
                                            '99.95th=[17280],\n'
                                            '     | 99.99th=[60672]\n'
                                            '   bw (  KiB/s): min=67403, '
                                            'max=67839, per=100.00%, '
                                            'avg=67627.79, stdev=111.23, '
                                            'samples=14\n'
                                            '   iops        : min=134806, '
                                            'max=135679, avg=135255.64, '
                                            'stdev=222.60, samples=14\n'
                                            '  lat (usec)   : 10=98.38%, '
                                            '20=1.60%, 50=0.01%, 100=0.02%\n'
                                            '  cpu          : usr=53.01%, '
                                            'sys=46.96%, ctx=10, majf=0, '
                                            'minf=0\n'
                                            '  IO depths    : 1=100.0%, '
                                            '2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, >=64=0.0%\n'
                                            '     submit    : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     complete  : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     issued rwts: '
                                            'total=946413,0,0,0 short=0,0,0,0 '
                                            'dropped=0,0,0,0\n'
                                            '     latency   : target=0, '
                                            'window=0, percentile=100.00%, '
                                            'depth=1\n'
                                            '\n'
                                            'Run status group 0 (all jobs):\n'
                                            '   READ: bw=66.0MiB/s (69.2MB/s), '
                                            '66.0MiB/s-66.0MiB/s '
                                            '(69.2MB/s-69.2MB/s), io=462MiB '
                                            '(485MB), run=7001-7001msec\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_39.output'),
                                  'state': {'begin': 1666782229.8945956,
                                            'cmd': 'cat '
                                                   '/tmp/fio-output_512_1_io_uring-bdev_uring_0.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.827765703201294,
                                            'end': 1666782230.7223613,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_39.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_39.state')},
                       'cmd_390': {'output': 'libaio-xnvme: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=xnvme, iodepth=8\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666791053,\n'
                                             '  "timestamp_ms" : '
                                             '1666791053029,\n'
                                             '  "time" : "Wed Oct 26 15:30:53 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"libaio-xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"libaio-xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "8",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
                                             '        "xnvme_be" : "linux",\n'
                                             '        "xnvme_async" : '
                                             '"libaio",\n'
                                             '        "xnvme_sync" : "nvme",\n'
                                             '        "xnvme_admin" : "nvme",\n'
                                             '        "xnvme_dev_nsid" : "1",\n'
                                             '        "filename" : '
                                             '"/dev/nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '1495874560,\n'
                                             '        "io_kbytes" : 1460815,\n'
                                             '        "bw_bytes" : 213665842,\n'
                                             '        "bw" : 208658,\n'
                                             '        "iops" : 417315.097843,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 2921623,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1512,\n'
                                             '          "max" : 36950,\n'
                                             '          "mean" : 1657.647678,\n'
                                             '          "stddev" : '
                                             '106.702955,\n'
                                             '          "N" : 2921623\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 5039,\n'
                                             '          "max" : 134552,\n'
                                             '          "mean" : '
                                             '17269.205602,\n'
                                             '          "stddev" : '
                                             '2767.694361,\n'
                                             '          "N" : 2921630,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 14912,\n'
                                             '            "5.000000" : 16512,\n'
                                             '            "10.000000" : '
                                             '16512,\n'
                                             '            "20.000000" : '
                                             '16768,\n'
                                             '            "30.000000" : '
                                             '16768,\n'
                                             '            "40.000000" : '
                                             '16768,\n'
                                             '            "50.000000" : '
                                             '16768,\n'
                                             '            "60.000000" : '
                                             '16768,\n'
                                             '            "70.000000" : '
                                             '16768,\n'
                                             '            "80.000000" : '
                                             '17280,\n'
                                             '            "90.000000" : '
                                             '17536,\n'
                                             '            "95.000000" : '
                                             '18048,\n'
                                             '            "99.000000" : '
                                             '30336,\n'
                                             '            "99.500000" : '
                                             '34560,\n'
                                             '            "99.900000" : '
                                             '50944,\n'
                                             '            "99.950000" : '
                                             '67072,\n'
                                             '            "99.990000" : 80384\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 6592,\n'
                                             '          "max" : 136265,\n'
                                             '          "mean" : '
                                             '18926.853252,\n'
                                             '          "stddev" : '
                                             '2770.487130,\n'
                                             '          "N" : 2921630\n'
                                             '        },\n'
                                             '        "bw_min" : 208125,\n'
                                             '        "bw_max" : 209317,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '208769.428571,\n'
                                             '        "bw_dev" : 418.942389,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 416250,\n'
                                             '        "iops_max" : 418634,\n'
                                             '        "iops_mean" : '
                                             '417538.857143,\n'
                                             '        "iops_stddev" : '
                                             '837.884778,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 34.928571,\n'
                                             '      "sys_cpu" : 63.642857,\n'
                                             '      "ctx" : 16304,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 100.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 99.999983,\n'
                                             '        "8" : 0.100000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.213888,\n'
                                             '        "20" : 96.208785,\n'
                                             '        "50" : 3.475089,\n'
                                             '        "100" : 0.101450,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 8,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'libaio-xnvme: (groupid=0, '
                                             'jobs=1): err= 0: pid=11651: Wed '
                                             'Oct 26 15:30:53 2022\n'
                                             '  read: IOPS=417k, BW=204MiB/s '
                                             '(214MB/s)(1427MiB/7001msec)\n'
                                             '    slat (nsec): min=1512, '
                                             'max=36950, avg=1657.65, '
                                             'stdev=106.70\n'
                                             '    clat (usec): min=5, max=134, '
                                             'avg=17.27, stdev= 2.77\n'
                                             '     lat (usec): min=6, max=136, '
                                             'avg=18.93, stdev= 2.77\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[14912],  '
                                             '5.00th=[16512], 10.00th=[16512], '
                                             '20.00th=[16768],\n'
                                             '     | 30.00th=[16768], '
                                             '40.00th=[16768], '
                                             '50.00th=[16768], '
                                             '60.00th=[16768],\n'
                                             '     | 70.00th=[16768], '
                                             '80.00th=[17280], '
                                             '90.00th=[17536], '
                                             '95.00th=[18048],\n'
                                             '     | 99.00th=[30336], '
                                             '99.50th=[34560], '
                                             '99.90th=[50944], '
                                             '99.95th=[67072],\n'
                                             '     | 99.99th=[80384]\n'
                                             '   bw (  KiB/s): min=208125, '
                                             'max=209317, per=100.00%, '
                                             'avg=208769.43, stdev=418.94, '
                                             'samples=14\n'
                                             '   iops        : min=416250, '
                                             'max=418634, avg=417538.86, '
                                             'stdev=837.88, samples=14\n'
                                             '  lat (usec)   : 10=0.21%, '
                                             '20=96.21%, 50=3.48%, 100=0.10%, '
                                             '250=0.01%\n'
                                             '  cpu          : usr=34.93%, '
                                             'sys=63.64%, ctx=16304, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, 2=0.0%, '
                                             '4=0.0%, 8=100.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.1%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=2921623,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=8\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=204MiB/s (214MB/s), '
                                             '204MiB/s-204MiB/s '
                                             '(214MB/s-214MB/s), io=1427MiB '
                                             '(1496MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_390.output'),
                                   'state': {'begin': 1666783962.9672246,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_8_libaio-xnvme_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8798742294311523,
                                             'end': 1666783963.8470988,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_390.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_390.state')},
                       'cmd_391': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_512_8_libaio-xnvme_1.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_391.output'),
                                   'state': {'begin': 1666783964.928627,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_8_libaio-xnvme_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8847477436065674,
                                             'end': 1666783965.8133748,
                                             'err': 1,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_391.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_391.state')},
                       'cmd_392': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_392.output'),
                                   'state': {'begin': 1666783965.8152633,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="libaio-xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="8" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
                                                    '--xnvme_be="linux" '
                                                    '--xnvme_async="libaio" '
                                                    '--xnvme_sync="nvme" '
                                                    '--xnvme_admin="nvme" '
                                                    '--xnvme_dev_nsid="1" '
                                                    '--filename="/dev/nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_8_libaio-xnvme_1.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.087571859359741,
                                             'end': 1666783976.9028351,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_392.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_392.state')},
                       'cmd_393': {'output': 'libaio-xnvme: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=xnvme, iodepth=8\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666791066,\n'
                                             '  "timestamp_ms" : '
                                             '1666791066967,\n'
                                             '  "time" : "Wed Oct 26 15:31:06 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"libaio-xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"libaio-xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "8",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
                                             '        "xnvme_be" : "linux",\n'
                                             '        "xnvme_async" : '
                                             '"libaio",\n'
                                             '        "xnvme_sync" : "nvme",\n'
                                             '        "xnvme_admin" : "nvme",\n'
                                             '        "xnvme_dev_nsid" : "1",\n'
                                             '        "filename" : '
                                             '"/dev/nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '1492745216,\n'
                                             '        "io_kbytes" : 1457759,\n'
                                             '        "bw_bytes" : 213218856,\n'
                                             '        "bw" : 208221,\n'
                                             '        "iops" : 416442.079703,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 2915511,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1502,\n'
                                             '          "max" : 33263,\n'
                                             '          "mean" : 1671.443877,\n'
                                             '          "stddev" : '
                                             '126.111968,\n'
                                             '          "N" : 2915511\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 5090,\n'
                                             '          "max" : 134011,\n'
                                             '          "mean" : '
                                             '17299.230179,\n'
                                             '          "stddev" : '
                                             '2704.916413,\n'
                                             '          "N" : 2915518,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 15040,\n'
                                             '            "5.000000" : 16512,\n'
                                             '            "10.000000" : '
                                             '16512,\n'
                                             '            "20.000000" : '
                                             '16768,\n'
                                             '            "30.000000" : '
                                             '16768,\n'
                                             '            "40.000000" : '
                                             '16768,\n'
                                             '            "50.000000" : '
                                             '16768,\n'
                                             '            "60.000000" : '
                                             '16768,\n'
                                             '            "70.000000" : '
                                             '17024,\n'
                                             '            "80.000000" : '
                                             '17280,\n'
                                             '            "90.000000" : '
                                             '17536,\n'
                                             '            "95.000000" : '
                                             '18048,\n'
                                             '            "99.000000" : '
                                             '30336,\n'
                                             '            "99.500000" : '
                                             '34560,\n'
                                             '            "99.900000" : '
                                             '47360,\n'
                                             '            "99.950000" : '
                                             '64256,\n'
                                             '            "99.990000" : 79360\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 6653,\n'
                                             '          "max" : 135624,\n'
                                             '          "mean" : '
                                             '18970.673988,\n'
                                             '          "stddev" : '
                                             '2707.653440,\n'
                                             '          "N" : 2915518\n'
                                             '        },\n'
                                             '        "bw_min" : 207578,\n'
                                             '        "bw_max" : 209884,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '208259.785714,\n'
                                             '        "bw_dev" : 724.737107,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 415156,\n'
                                             '        "iops_max" : 419769,\n'
                                             '        "iops_mean" : '
                                             '416519.500000,\n'
                                             '        "iops_stddev" : '
                                             '1449.767660,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 30.614286,\n'
                                             '      "sys_cpu" : 67.985714,\n'
                                             '      "ctx" : 16266,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 100.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 99.999983,\n'
                                             '        "8" : 0.100000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.206173,\n'
                                             '        "20" : 96.180395,\n'
                                             '        "50" : 3.523842,\n'
                                             '        "100" : 0.088767,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 8,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'libaio-xnvme: (groupid=0, '
                                             'jobs=1): err= 0: pid=11700: Wed '
                                             'Oct 26 15:31:06 2022\n'
                                             '  read: IOPS=416k, BW=203MiB/s '
                                             '(213MB/s)(1424MiB/7001msec)\n'
                                             '    slat (nsec): min=1502, '
                                             'max=33263, avg=1671.44, '
                                             'stdev=126.11\n'
                                             '    clat (usec): min=5, max=134, '
                                             'avg=17.30, stdev= 2.70\n'
                                             '     lat (usec): min=6, max=135, '
                                             'avg=18.97, stdev= 2.71\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[15040],  '
                                             '5.00th=[16512], 10.00th=[16512], '
                                             '20.00th=[16768],\n'
                                             '     | 30.00th=[16768], '
                                             '40.00th=[16768], '
                                             '50.00th=[16768], '
                                             '60.00th=[16768],\n'
                                             '     | 70.00th=[17024], '
                                             '80.00th=[17280], '
                                             '90.00th=[17536], '
                                             '95.00th=[18048],\n'
                                             '     | 99.00th=[30336], '
                                             '99.50th=[34560], '
                                             '99.90th=[47360], '
                                             '99.95th=[64256],\n'
                                             '     | 99.99th=[79360]\n'
                                             '   bw (  KiB/s): min=207578, '
                                             'max=209884, per=100.00%, '
                                             'avg=208259.79, stdev=724.74, '
                                             'samples=14\n'
                                             '   iops        : min=415156, '
                                             'max=419769, avg=416519.50, '
                                             'stdev=1449.77, samples=14\n'
                                             '  lat (usec)   : 10=0.21%, '
                                             '20=96.18%, 50=3.52%, 100=0.09%, '
                                             '250=0.01%\n'
                                             '  cpu          : usr=30.61%, '
                                             'sys=67.99%, ctx=16266, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, 2=0.0%, '
                                             '4=0.0%, 8=100.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.1%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=2915511,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=8\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=203MiB/s (213MB/s), '
                                             '203MiB/s-203MiB/s '
                                             '(213MB/s-213MB/s), io=1424MiB '
                                             '(1493MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_393.output'),
                                   'state': {'begin': 1666783976.9056933,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_8_libaio-xnvme_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.856008768081665,
                                             'end': 1666783977.761702,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_393.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_393.state')},
                       'cmd_394': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_512_8_libaio-xnvme_2.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_394.output'),
                                   'state': {'begin': 1666783978.8074331,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_8_libaio-xnvme_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8541994094848633,
                                             'end': 1666783979.6616325,
                                             'err': 1,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_394.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_394.state')},
                       'cmd_395': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_395.output'),
                                   'state': {'begin': 1666783979.6635144,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="libaio-xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="8" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
                                                    '--xnvme_be="linux" '
                                                    '--xnvme_async="libaio" '
                                                    '--xnvme_sync="nvme" '
                                                    '--xnvme_admin="nvme" '
                                                    '--xnvme_dev_nsid="1" '
                                                    '--filename="/dev/nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_8_libaio-xnvme_2.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.125166177749634,
                                             'end': 1666783990.7886806,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_395.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_395.state')},
                       'cmd_396': {'output': 'libaio-xnvme: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=xnvme, iodepth=8\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666791080,\n'
                                             '  "timestamp_ms" : '
                                             '1666791080852,\n'
                                             '  "time" : "Wed Oct 26 15:31:20 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"libaio-xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"libaio-xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "8",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
                                             '        "xnvme_be" : "linux",\n'
                                             '        "xnvme_async" : '
                                             '"libaio",\n'
                                             '        "xnvme_sync" : "nvme",\n'
                                             '        "xnvme_admin" : "nvme",\n'
                                             '        "xnvme_dev_nsid" : "1",\n'
                                             '        "filename" : '
                                             '"/dev/nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '1497201664,\n'
                                             '        "io_kbytes" : 1462111,\n'
                                             '        "bw_bytes" : 213855401,\n'
                                             '        "bw" : 208843,\n'
                                             '        "iops" : 417685.187830,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 2924214,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1513,\n'
                                             '          "max" : 25247,\n'
                                             '          "mean" : 1654.304958,\n'
                                             '          "stddev" : '
                                             '100.956893,\n'
                                             '          "N" : 2924215\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 5080,\n'
                                             '          "max" : 137948,\n'
                                             '          "mean" : '
                                             '17253.342018,\n'
                                             '          "stddev" : '
                                             '2675.005836,\n'
                                             '          "N" : 2924221,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 14784,\n'
                                             '            "5.000000" : 16768,\n'
                                             '            "10.000000" : '
                                             '16768,\n'
                                             '            "20.000000" : '
                                             '16768,\n'
                                             '            "30.000000" : '
                                             '16768,\n'
                                             '            "40.000000" : '
                                             '16768,\n'
                                             '            "50.000000" : '
                                             '16768,\n'
                                             '            "60.000000" : '
                                             '16768,\n'
                                             '            "70.000000" : '
                                             '17024,\n'
                                             '            "80.000000" : '
                                             '17024,\n'
                                             '            "90.000000" : '
                                             '17536,\n'
                                             '            "95.000000" : '
                                             '17792,\n'
                                             '            "99.000000" : '
                                             '29568,\n'
                                             '            "99.500000" : '
                                             '33536,\n'
                                             '            "99.900000" : '
                                             '50944,\n'
                                             '            "99.950000" : '
                                             '66048,\n'
                                             '            "99.990000" : 79360\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 6703,\n'
                                             '          "max" : 139541,\n'
                                             '          "mean" : '
                                             '18907.646970,\n'
                                             '          "stddev" : '
                                             '2674.174544,\n'
                                             '          "N" : 2924221\n'
                                             '        },\n'
                                             '        "bw_min" : 208370,\n'
                                             '        "bw_max" : 209183,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '208880.357143,\n'
                                             '        "bw_dev" : 273.334849,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 416741,\n'
                                             '        "iops_max" : 418366,\n'
                                             '        "iops_mean" : '
                                             '417760.785714,\n'
                                             '        "iops_stddev" : '
                                             '546.526117,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 32.342857,\n'
                                             '      "sys_cpu" : 66.300000,\n'
                                             '      "ctx" : 15925,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 100.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 99.999983,\n'
                                             '        "8" : 0.100000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.234285,\n'
                                             '        "20" : 96.261628,\n'
                                             '        "50" : 3.402384,\n'
                                             '        "100" : 0.100574,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 8,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'libaio-xnvme: (groupid=0, '
                                             'jobs=1): err= 0: pid=11749: Wed '
                                             'Oct 26 15:31:20 2022\n'
                                             '  read: IOPS=418k, BW=204MiB/s '
                                             '(214MB/s)(1428MiB/7001msec)\n'
                                             '    slat (nsec): min=1513, '
                                             'max=25247, avg=1654.30, '
                                             'stdev=100.96\n'
                                             '    clat (usec): min=5, max=137, '
                                             'avg=17.25, stdev= 2.68\n'
                                             '     lat (usec): min=6, max=139, '
                                             'avg=18.91, stdev= 2.67\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[14784],  '
                                             '5.00th=[16768], 10.00th=[16768], '
                                             '20.00th=[16768],\n'
                                             '     | 30.00th=[16768], '
                                             '40.00th=[16768], '
                                             '50.00th=[16768], '
                                             '60.00th=[16768],\n'
                                             '     | 70.00th=[17024], '
                                             '80.00th=[17024], '
                                             '90.00th=[17536], '
                                             '95.00th=[17792],\n'
                                             '     | 99.00th=[29568], '
                                             '99.50th=[33536], '
                                             '99.90th=[50944], '
                                             '99.95th=[66048],\n'
                                             '     | 99.99th=[79360]\n'
                                             '   bw (  KiB/s): min=208370, '
                                             'max=209183, per=100.00%, '
                                             'avg=208880.36, stdev=273.33, '
                                             'samples=14\n'
                                             '   iops        : min=416741, '
                                             'max=418366, avg=417760.79, '
                                             'stdev=546.53, samples=14\n'
                                             '  lat (usec)   : 10=0.23%, '
                                             '20=96.26%, 50=3.40%, 100=0.10%, '
                                             '250=0.01%\n'
                                             '  cpu          : usr=32.34%, '
                                             'sys=66.30%, ctx=15925, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, 2=0.0%, '
                                             '4=0.0%, 8=100.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.1%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=2924214,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=8\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=204MiB/s (214MB/s), '
                                             '204MiB/s-204MiB/s '
                                             '(214MB/s-214MB/s), io=1428MiB '
                                             '(1497MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_396.output'),
                                   'state': {'begin': 1666783990.7907233,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_8_libaio-xnvme_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.846900224685669,
                                             'end': 1666783991.6376235,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_396.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_396.state')},
                       'cmd_40': {'output': 'rm: cannot remove '
                                            "'/tmp/fio-output_512_1_io_uring-bdev_uring_1.txt': "
                                            'No such file or directory\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_40.output'),
                                  'state': {'begin': 1666782232.8952932,
                                            'cmd': 'rm '
                                                   '/tmp/fio-output_512_1_io_uring-bdev_uring_1.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.8036515712738037,
                                            'end': 1666782233.6989448,
                                            'err': 1,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_40.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_40.state')},
                       'cmd_41': {'output': 'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_41.output'),
                                  'state': {'begin': 1666782233.7012796,
                                            'cmd': '/opt/fio/bin/fio '
                                                   '--name="io_uring-bdev_uring" '
                                                   '--rw="randread" '
                                                   '--size="12G" --iodepth="1" '
                                                   '--bs="512" --direct="1" '
                                                   '--thread="1" '
                                                   '--time_based="1" '
                                                   '--runtime="7" '
                                                   '--ramp_time="3" '
                                                   '--norandommap="1" '
                                                   '--allow_file_create="0" '
                                                   '--ioengine="spdk_bdev" '
                                                   '--spdk_conf="/tmp/spdk_bdev_uring_foo.conf" '
                                                   '--filename="Nvme0n1" '
                                                   '--output-format="normal,json" '
                                                   '--output="/tmp/fio-output_512_1_io_uring-bdev_uring_1.txt" '
                                                   '--cpus_allowed="1"',
                                            'cwd': 'None',
                                            'elapsed': 11.568262815475464,
                                            'end': 1666782245.2695425,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_41.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_41.state')},
                       'cmd_42': {'output': 'io_uring-bdev_uring: (g=0): '
                                            'rw=randread, bs=(R) 512B-512B, '
                                            '(W) 512B-512B, (T) 512B-512B, '
                                            'ioengine=spdk_bdev, iodepth=1\n'
                                            'fio-3.32-dirty\n'
                                            'Starting 1 thread\n'
                                            '{\n'
                                            '  "fio version" : '
                                            '"fio-3.32-dirty",\n'
                                            '  "timestamp" : 1666789335,\n'
                                            '  "timestamp_ms" : '
                                            '1666789335250,\n'
                                            '  "time" : "Wed Oct 26 15:02:15 '
                                            '2022",\n'
                                            '  "jobs" : [\n'
                                            '    {\n'
                                            '      "jobname" : '
                                            '"io_uring-bdev_uring",\n'
                                            '      "groupid" : 0,\n'
                                            '      "error" : 0,\n'
                                            '      "eta" : 0,\n'
                                            '      "elapsed" : 11,\n'
                                            '      "job options" : {\n'
                                            '        "name" : '
                                            '"io_uring-bdev_uring",\n'
                                            '        "rw" : "randread",\n'
                                            '        "size" : "12G",\n'
                                            '        "iodepth" : "1",\n'
                                            '        "bs" : "512",\n'
                                            '        "direct" : "1",\n'
                                            '        "thread" : "1",\n'
                                            '        "time_based" : "1",\n'
                                            '        "runtime" : "7",\n'
                                            '        "ramp_time" : "3",\n'
                                            '        "norandommap" : "1",\n'
                                            '        "allow_file_create" : '
                                            '"0",\n'
                                            '        "ioengine" : '
                                            '"spdk_bdev",\n'
                                            '        "spdk_conf" : '
                                            '"/tmp/spdk_bdev_uring_foo.conf",\n'
                                            '        "filename" : "Nvme0n1",\n'
                                            '        "cpus_allowed" : "1"\n'
                                            '      },\n'
                                            '      "read" : {\n'
                                            '        "io_bytes" : 483598848,\n'
                                            '        "io_kbytes" : 472264,\n'
                                            '        "bw_bytes" : 69075681,\n'
                                            '        "bw" : 67456,\n'
                                            '        "iops" : 134913.440937,\n'
                                            '        "runtime" : 7001,\n'
                                            '        "total_ios" : 944529,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 160,\n'
                                            '          "max" : 10830,\n'
                                            '          "mean" : 176.276544,\n'
                                            '          "stddev" : 42.021704,\n'
                                            '          "N" : 944529\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 6261,\n'
                                            '          "max" : 115436,\n'
                                            '          "mean" : 6989.813789,\n'
                                            '          "stddev" : '
                                            '1151.700203,\n'
                                            '          "N" : 944529,\n'
                                            '          "percentile" : {\n'
                                            '            "1.000000" : 6496,\n'
                                            '            "5.000000" : 6688,\n'
                                            '            "10.000000" : 6752,\n'
                                            '            "20.000000" : 6752,\n'
                                            '            "30.000000" : 6752,\n'
                                            '            "40.000000" : 6816,\n'
                                            '            "50.000000" : 6816,\n'
                                            '            "60.000000" : 6880,\n'
                                            '            "70.000000" : 6880,\n'
                                            '            "80.000000" : 6944,\n'
                                            '            "90.000000" : 7008,\n'
                                            '            "95.000000" : 7136,\n'
                                            '            "99.000000" : 13120,\n'
                                            '            "99.500000" : 14656,\n'
                                            '            "99.900000" : 17024,\n'
                                            '            "99.950000" : 17280,\n'
                                            '            "99.990000" : 58624\n'
                                            '          }\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 6432,\n'
                                            '          "max" : 115607,\n'
                                            '          "mean" : 7166.090333,\n'
                                            '          "stddev" : '
                                            '1152.795263,\n'
                                            '          "N" : 944529\n'
                                            '        },\n'
                                            '        "bw_min" : 67337,\n'
                                            '        "bw_max" : 67822,\n'
                                            '        "bw_agg" : 100.000000,\n'
                                            '        "bw_mean" : '
                                            '67490.142857,\n'
                                            '        "bw_dev" : 143.619614,\n'
                                            '        "bw_samples" : 14,\n'
                                            '        "iops_min" : 134674,\n'
                                            '        "iops_max" : 135645,\n'
                                            '        "iops_mean" : '
                                            '134980.500000,\n'
                                            '        "iops_stddev" : '
                                            '287.344272,\n'
                                            '        "iops_samples" : 14\n'
                                            '      },\n'
                                            '      "write" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "trim" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "sync" : {\n'
                                            '        "total_ios" : 0,\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        }\n'
                                            '      },\n'
                                            '      "job_runtime" : 7000,\n'
                                            '      "usr_cpu" : 54.442857,\n'
                                            '      "sys_cpu" : 45.528571,\n'
                                            '      "ctx" : 10,\n'
                                            '      "majf" : 0,\n'
                                            '      "minf" : 0,\n'
                                            '      "iodepth_level" : {\n'
                                            '        "1" : 100.000000,\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_submit" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_complete" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ns" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_us" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 98.367546,\n'
                                            '        "20" : 1.614985,\n'
                                            '        "50" : 0.010000,\n'
                                            '        "100" : 0.013128,\n'
                                            '        "250" : 0.010000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ms" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000,\n'
                                            '        "2000" : 0.000000,\n'
                                            '        ">=2000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_depth" : 1,\n'
                                            '      "latency_target" : 0,\n'
                                            '      "latency_percentile" : '
                                            '100.000000,\n'
                                            '      "latency_window" : 0\n'
                                            '    }\n'
                                            '  ]\n'
                                            '}\n'
                                            '\n'
                                            'io_uring-bdev_uring: (groupid=0, '
                                            'jobs=1): err= 0: pid=5597: Wed '
                                            'Oct 26 15:02:15 2022\n'
                                            '  read: IOPS=135k, BW=65.9MiB/s '
                                            '(69.1MB/s)(461MiB/7001msec)\n'
                                            '    slat (nsec): min=160, '
                                            'max=10830, avg=176.28, '
                                            'stdev=42.02\n'
                                            '    clat (usec): min=6, max=115, '
                                            'avg= 6.99, stdev= 1.15\n'
                                            '     lat (usec): min=6, max=115, '
                                            'avg= 7.17, stdev= 1.15\n'
                                            '    clat percentiles (nsec):\n'
                                            '     |  1.00th=[ 6496],  5.00th=[ '
                                            '6688], 10.00th=[ 6752], 20.00th=[ '
                                            '6752],\n'
                                            '     | 30.00th=[ 6752], 40.00th=[ '
                                            '6816], 50.00th=[ 6816], 60.00th=[ '
                                            '6880],\n'
                                            '     | 70.00th=[ 6880], 80.00th=[ '
                                            '6944], 90.00th=[ 7008], 95.00th=[ '
                                            '7136],\n'
                                            '     | 99.00th=[13120], '
                                            '99.50th=[14656], 99.90th=[17024], '
                                            '99.95th=[17280],\n'
                                            '     | 99.99th=[58624]\n'
                                            '   bw (  KiB/s): min=67337, '
                                            'max=67822, per=100.00%, '
                                            'avg=67490.14, stdev=143.62, '
                                            'samples=14\n'
                                            '   iops        : min=134674, '
                                            'max=135645, avg=134980.50, '
                                            'stdev=287.34, samples=14\n'
                                            '  lat (usec)   : 10=98.37%, '
                                            '20=1.61%, 50=0.01%, 100=0.01%, '
                                            '250=0.01%\n'
                                            '  cpu          : usr=54.44%, '
                                            'sys=45.53%, ctx=10, majf=0, '
                                            'minf=0\n'
                                            '  IO depths    : 1=100.0%, '
                                            '2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, >=64=0.0%\n'
                                            '     submit    : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     complete  : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     issued rwts: '
                                            'total=944529,0,0,0 short=0,0,0,0 '
                                            'dropped=0,0,0,0\n'
                                            '     latency   : target=0, '
                                            'window=0, percentile=100.00%, '
                                            'depth=1\n'
                                            '\n'
                                            'Run status group 0 (all jobs):\n'
                                            '   READ: bw=65.9MiB/s (69.1MB/s), '
                                            '65.9MiB/s-65.9MiB/s '
                                            '(69.1MB/s-69.1MB/s), io=461MiB '
                                            '(484MB), run=7001-7001msec\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_42.output'),
                                  'state': {'begin': 1666782245.2729244,
                                            'cmd': 'cat '
                                                   '/tmp/fio-output_512_1_io_uring-bdev_uring_1.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.8888406753540039,
                                            'end': 1666782246.161765,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_42.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_42.state')},
                       'cmd_43': {'output': 'rm: cannot remove '
                                            "'/tmp/fio-output_512_1_io_uring-bdev_uring_2.txt': "
                                            'No such file or directory\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_43.output'),
                                  'state': {'begin': 1666782248.3104722,
                                            'cmd': 'rm '
                                                   '/tmp/fio-output_512_1_io_uring-bdev_uring_2.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.8213715553283691,
                                            'end': 1666782249.1318438,
                                            'err': 1,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_43.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_43.state')},
                       'cmd_44': {'output': 'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_44.output'),
                                  'state': {'begin': 1666782249.1338692,
                                            'cmd': '/opt/fio/bin/fio '
                                                   '--name="io_uring-bdev_uring" '
                                                   '--rw="randread" '
                                                   '--size="12G" --iodepth="1" '
                                                   '--bs="512" --direct="1" '
                                                   '--thread="1" '
                                                   '--time_based="1" '
                                                   '--runtime="7" '
                                                   '--ramp_time="3" '
                                                   '--norandommap="1" '
                                                   '--allow_file_create="0" '
                                                   '--ioengine="spdk_bdev" '
                                                   '--spdk_conf="/tmp/spdk_bdev_uring_foo.conf" '
                                                   '--filename="Nvme0n1" '
                                                   '--output-format="normal,json" '
                                                   '--output="/tmp/fio-output_512_1_io_uring-bdev_uring_2.txt" '
                                                   '--cpus_allowed="1"',
                                            'cwd': 'None',
                                            'elapsed': 11.446779727935791,
                                            'end': 1666782260.580649,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_44.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_44.state')},
                       'cmd_45': {'output': 'io_uring-bdev_uring: (g=0): '
                                            'rw=randread, bs=(R) 512B-512B, '
                                            '(W) 512B-512B, (T) 512B-512B, '
                                            'ioengine=spdk_bdev, iodepth=1\n'
                                            'fio-3.32-dirty\n'
                                            'Starting 1 thread\n'
                                            '{\n'
                                            '  "fio version" : '
                                            '"fio-3.32-dirty",\n'
                                            '  "timestamp" : 1666789350,\n'
                                            '  "timestamp_ms" : '
                                            '1666789350610,\n'
                                            '  "time" : "Wed Oct 26 15:02:30 '
                                            '2022",\n'
                                            '  "jobs" : [\n'
                                            '    {\n'
                                            '      "jobname" : '
                                            '"io_uring-bdev_uring",\n'
                                            '      "groupid" : 0,\n'
                                            '      "error" : 0,\n'
                                            '      "eta" : 0,\n'
                                            '      "elapsed" : 11,\n'
                                            '      "job options" : {\n'
                                            '        "name" : '
                                            '"io_uring-bdev_uring",\n'
                                            '        "rw" : "randread",\n'
                                            '        "size" : "12G",\n'
                                            '        "iodepth" : "1",\n'
                                            '        "bs" : "512",\n'
                                            '        "direct" : "1",\n'
                                            '        "thread" : "1",\n'
                                            '        "time_based" : "1",\n'
                                            '        "runtime" : "7",\n'
                                            '        "ramp_time" : "3",\n'
                                            '        "norandommap" : "1",\n'
                                            '        "allow_file_create" : '
                                            '"0",\n'
                                            '        "ioengine" : '
                                            '"spdk_bdev",\n'
                                            '        "spdk_conf" : '
                                            '"/tmp/spdk_bdev_uring_foo.conf",\n'
                                            '        "filename" : "Nvme0n1",\n'
                                            '        "cpus_allowed" : "1"\n'
                                            '      },\n'
                                            '      "read" : {\n'
                                            '        "io_bytes" : 482136064,\n'
                                            '        "io_kbytes" : 470836,\n'
                                            '        "bw_bytes" : 68866742,\n'
                                            '        "bw" : 67252,\n'
                                            '        "iops" : 134505.356378,\n'
                                            '        "runtime" : 7001,\n'
                                            '        "total_ios" : 941672,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 160,\n'
                                            '          "max" : 9859,\n'
                                            '          "mean" : 177.184497,\n'
                                            '          "stddev" : 34.333593,\n'
                                            '          "N" : 941672\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 6301,\n'
                                            '          "max" : 114083,\n'
                                            '          "mean" : 7024.269096,\n'
                                            '          "stddev" : '
                                            '1168.593145,\n'
                                            '          "N" : 941672,\n'
                                            '          "percentile" : {\n'
                                            '            "1.000000" : 6560,\n'
                                            '            "5.000000" : 6752,\n'
                                            '            "10.000000" : 6752,\n'
                                            '            "20.000000" : 6816,\n'
                                            '            "30.000000" : 6816,\n'
                                            '            "40.000000" : 6816,\n'
                                            '            "50.000000" : 6880,\n'
                                            '            "60.000000" : 6880,\n'
                                            '            "70.000000" : 6944,\n'
                                            '            "80.000000" : 6944,\n'
                                            '            "90.000000" : 7072,\n'
                                            '            "95.000000" : 7136,\n'
                                            '            "99.000000" : 13120,\n'
                                            '            "99.500000" : 14656,\n'
                                            '            "99.900000" : 17024,\n'
                                            '            "99.950000" : 17280,\n'
                                            '            "99.990000" : 59136\n'
                                            '          }\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 6472,\n'
                                            '          "max" : 114264,\n'
                                            '          "mean" : 7201.453593,\n'
                                            '          "stddev" : '
                                            '1169.346311,\n'
                                            '          "N" : 941672\n'
                                            '        },\n'
                                            '        "bw_min" : 67168,\n'
                                            '        "bw_max" : 67432,\n'
                                            '        "bw_agg" : 100.000000,\n'
                                            '        "bw_mean" : '
                                            '67286.928571,\n'
                                            '        "bw_dev" : 82.405624,\n'
                                            '        "bw_samples" : 14,\n'
                                            '        "iops_min" : 134336,\n'
                                            '        "iops_max" : 134865,\n'
                                            '        "iops_mean" : '
                                            '134574.000000,\n'
                                            '        "iops_stddev" : '
                                            '164.977854,\n'
                                            '        "iops_samples" : 14\n'
                                            '      },\n'
                                            '      "write" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "trim" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "sync" : {\n'
                                            '        "total_ios" : 0,\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        }\n'
                                            '      },\n'
                                            '      "job_runtime" : 7000,\n'
                                            '      "usr_cpu" : 58.614286,\n'
                                            '      "sys_cpu" : 41.357143,\n'
                                            '      "ctx" : 10,\n'
                                            '      "majf" : 0,\n'
                                            '      "minf" : 0,\n'
                                            '      "iodepth_level" : {\n'
                                            '        "1" : 100.000000,\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_submit" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_complete" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ns" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_us" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 98.349638,\n'
                                            '        "20" : 1.631885,\n'
                                            '        "50" : 0.010000,\n'
                                            '        "100" : 0.014124,\n'
                                            '        "250" : 0.010000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ms" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000,\n'
                                            '        "2000" : 0.000000,\n'
                                            '        ">=2000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_depth" : 1,\n'
                                            '      "latency_target" : 0,\n'
                                            '      "latency_percentile" : '
                                            '100.000000,\n'
                                            '      "latency_window" : 0\n'
                                            '    }\n'
                                            '  ]\n'
                                            '}\n'
                                            '\n'
                                            'io_uring-bdev_uring: (groupid=0, '
                                            'jobs=1): err= 0: pid=5654: Wed '
                                            'Oct 26 15:02:30 2022\n'
                                            '  read: IOPS=135k, BW=65.7MiB/s '
                                            '(68.9MB/s)(460MiB/7001msec)\n'
                                            '    slat (nsec): min=160, '
                                            'max=9859, avg=177.18, '
                                            'stdev=34.33\n'
                                            '    clat (usec): min=6, max=114, '
                                            'avg= 7.02, stdev= 1.17\n'
                                            '     lat (usec): min=6, max=114, '
                                            'avg= 7.20, stdev= 1.17\n'
                                            '    clat percentiles (nsec):\n'
                                            '     |  1.00th=[ 6560],  5.00th=[ '
                                            '6752], 10.00th=[ 6752], 20.00th=[ '
                                            '6816],\n'
                                            '     | 30.00th=[ 6816], 40.00th=[ '
                                            '6816], 50.00th=[ 6880], 60.00th=[ '
                                            '6880],\n'
                                            '     | 70.00th=[ 6944], 80.00th=[ '
                                            '6944], 90.00th=[ 7072], 95.00th=[ '
                                            '7136],\n'
                                            '     | 99.00th=[13120], '
                                            '99.50th=[14656], 99.90th=[17024], '
                                            '99.95th=[17280],\n'
                                            '     | 99.99th=[59136]\n'
                                            '   bw (  KiB/s): min=67168, '
                                            'max=67432, per=100.00%, '
                                            'avg=67286.93, stdev=82.41, '
                                            'samples=14\n'
                                            '   iops        : min=134336, '
                                            'max=134865, avg=134574.00, '
                                            'stdev=164.98, samples=14\n'
                                            '  lat (usec)   : 10=98.35%, '
                                            '20=1.63%, 50=0.01%, 100=0.01%, '
                                            '250=0.01%\n'
                                            '  cpu          : usr=58.61%, '
                                            'sys=41.36%, ctx=10, majf=0, '
                                            'minf=0\n'
                                            '  IO depths    : 1=100.0%, '
                                            '2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, >=64=0.0%\n'
                                            '     submit    : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     complete  : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     issued rwts: '
                                            'total=941672,0,0,0 short=0,0,0,0 '
                                            'dropped=0,0,0,0\n'
                                            '     latency   : target=0, '
                                            'window=0, percentile=100.00%, '
                                            'depth=1\n'
                                            '\n'
                                            'Run status group 0 (all jobs):\n'
                                            '   READ: bw=65.7MiB/s (68.9MB/s), '
                                            '65.7MiB/s-65.7MiB/s '
                                            '(68.9MB/s-68.9MB/s), io=460MiB '
                                            '(482MB), run=7001-7001msec\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_45.output'),
                                  'state': {'begin': 1666782260.583536,
                                            'cmd': 'cat '
                                                   '/tmp/fio-output_512_1_io_uring-bdev_uring_2.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.8505508899688721,
                                            'end': 1666782261.4340868,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_45.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_45.state')},
                       'cmd_46': {'output': 'rm: cannot remove '
                                            "'/tmp/fio-output_512_1_io_uring-bdev_xnvme_0.txt': "
                                            'No such file or directory\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_46.output'),
                                  'state': {'begin': 1666782263.8989596,
                                            'cmd': 'rm '
                                                   '/tmp/fio-output_512_1_io_uring-bdev_xnvme_0.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.9638922214508057,
                                            'end': 1666782264.8628519,
                                            'err': 1,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_46.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_46.state')},
                       'cmd_47': {'output': 'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_47.output'),
                                  'state': {'begin': 1666782264.869519,
                                            'cmd': '/opt/fio/bin/fio '
                                                   '--name="io_uring-bdev_xnvme" '
                                                   '--rw="randread" '
                                                   '--size="12G" --iodepth="1" '
                                                   '--bs="512" --direct="1" '
                                                   '--thread="1" '
                                                   '--time_based="1" '
                                                   '--runtime="7" '
                                                   '--ramp_time="3" '
                                                   '--norandommap="1" '
                                                   '--allow_file_create="0" '
                                                   '--ioengine="spdk_bdev" '
                                                   '--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" '
                                                   '--filename="Nvme0n1" '
                                                   '--output-format="normal,json" '
                                                   '--output="/tmp/fio-output_512_1_io_uring-bdev_xnvme_0.txt" '
                                                   '--cpus_allowed="1"',
                                            'cwd': 'None',
                                            'elapsed': 11.649443626403809,
                                            'end': 1666782276.5189626,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_47.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_47.state')},
                       'cmd_48': {'output': 'io_uring-bdev_xnvme: (g=0): '
                                            'rw=randread, bs=(R) 512B-512B, '
                                            '(W) 512B-512B, (T) 512B-512B, '
                                            'ioengine=spdk_bdev, iodepth=1\n'
                                            'fio-3.32-dirty\n'
                                            'Starting 1 thread\n'
                                            '{\n'
                                            '  "fio version" : '
                                            '"fio-3.32-dirty",\n'
                                            '  "timestamp" : 1666789366,\n'
                                            '  "timestamp_ms" : '
                                            '1666789366545,\n'
                                            '  "time" : "Wed Oct 26 15:02:46 '
                                            '2022",\n'
                                            '  "jobs" : [\n'
                                            '    {\n'
                                            '      "jobname" : '
                                            '"io_uring-bdev_xnvme",\n'
                                            '      "groupid" : 0,\n'
                                            '      "error" : 0,\n'
                                            '      "eta" : 0,\n'
                                            '      "elapsed" : 11,\n'
                                            '      "job options" : {\n'
                                            '        "name" : '
                                            '"io_uring-bdev_xnvme",\n'
                                            '        "rw" : "randread",\n'
                                            '        "size" : "12G",\n'
                                            '        "iodepth" : "1",\n'
                                            '        "bs" : "512",\n'
                                            '        "direct" : "1",\n'
                                            '        "thread" : "1",\n'
                                            '        "time_based" : "1",\n'
                                            '        "runtime" : "7",\n'
                                            '        "ramp_time" : "3",\n'
                                            '        "norandommap" : "1",\n'
                                            '        "allow_file_create" : '
                                            '"0",\n'
                                            '        "ioengine" : '
                                            '"spdk_bdev",\n'
                                            '        "spdk_conf" : '
                                            '"/tmp/spdk_bdev_xnvme_io_uring.conf",\n'
                                            '        "filename" : "Nvme0n1",\n'
                                            '        "cpus_allowed" : "1"\n'
                                            '      },\n'
                                            '      "read" : {\n'
                                            '        "io_bytes" : 551760384,\n'
                                            '        "io_kbytes" : 538828,\n'
                                            '        "bw_bytes" : 78811653,\n'
                                            '        "bw" : 76964,\n'
                                            '        "iops" : 153928.867305,\n'
                                            '        "runtime" : 7001,\n'
                                            '        "total_ios" : 1077656,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 1673,\n'
                                            '          "max" : 19667,\n'
                                            '          "mean" : 1817.265533,\n'
                                            '          "stddev" : 149.648202,\n'
                                            '          "N" : 1077657\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 240,\n'
                                            '          "max" : 99747,\n'
                                            '          "mean" : 4449.052741,\n'
                                            '          "stddev" : '
                                            '1090.885033,\n'
                                            '          "N" : 1077656,\n'
                                            '          "percentile" : {\n'
                                            '            "1.000000" : 4016,\n'
                                            '            "5.000000" : 4192,\n'
                                            '            "10.000000" : 4256,\n'
                                            '            "20.000000" : 4256,\n'
                                            '            "30.000000" : 4256,\n'
                                            '            "40.000000" : 4320,\n'
                                            '            "50.000000" : 4320,\n'
                                            '            "60.000000" : 4320,\n'
                                            '            "70.000000" : 4384,\n'
                                            '            "80.000000" : 4384,\n'
                                            '            "90.000000" : 4448,\n'
                                            '            "95.000000" : 4512,\n'
                                            '            "99.000000" : 9664,\n'
                                            '            "99.500000" : 10688,\n'
                                            '            "99.900000" : 15296,\n'
                                            '            "99.950000" : 15424,\n'
                                            '            "99.990000" : 57088\n'
                                            '          }\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 5521,\n'
                                            '          "max" : 101600,\n'
                                            '          "mean" : 6266.318333,\n'
                                            '          "stddev" : '
                                            '1102.612816,\n'
                                            '          "N" : 1077656\n'
                                            '        },\n'
                                            '        "bw_min" : 76849,\n'
                                            '        "bw_max" : 77209,\n'
                                            '        "bw_agg" : 100.000000,\n'
                                            '        "bw_mean" : '
                                            '77006.785714,\n'
                                            '        "bw_dev" : 107.468907,\n'
                                            '        "bw_samples" : 14,\n'
                                            '        "iops_min" : 153699,\n'
                                            '        "iops_max" : 154418,\n'
                                            '        "iops_mean" : '
                                            '154013.714286,\n'
                                            '        "iops_stddev" : '
                                            '214.803104,\n'
                                            '        "iops_samples" : 14\n'
                                            '      },\n'
                                            '      "write" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "trim" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "sync" : {\n'
                                            '        "total_ios" : 0,\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        }\n'
                                            '      },\n'
                                            '      "job_runtime" : 7000,\n'
                                            '      "usr_cpu" : 16.271429,\n'
                                            '      "sys_cpu" : 83.714286,\n'
                                            '      "ctx" : 7,\n'
                                            '      "majf" : 0,\n'
                                            '      "minf" : 0,\n'
                                            '      "iodepth_level" : {\n'
                                            '        "1" : 100.000000,\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_submit" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_complete" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ns" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.010000,\n'
                                            '        "500" : 0.010000,\n'
                                            '        "750" : 0.010000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_us" : {\n'
                                            '        "2" : 0.010000,\n'
                                            '        "4" : 0.958284,\n'
                                            '        "10" : 98.165184,\n'
                                            '        "20" : 0.857880,\n'
                                            '        "50" : 0.010000,\n'
                                            '        "100" : 0.012249,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ms" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000,\n'
                                            '        "2000" : 0.000000,\n'
                                            '        ">=2000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_depth" : 1,\n'
                                            '      "latency_target" : 0,\n'
                                            '      "latency_percentile" : '
                                            '100.000000,\n'
                                            '      "latency_window" : 0\n'
                                            '    }\n'
                                            '  ]\n'
                                            '}\n'
                                            '\n'
                                            'io_uring-bdev_xnvme: (groupid=0, '
                                            'jobs=1): err= 0: pid=5712: Wed '
                                            'Oct 26 15:02:46 2022\n'
                                            '  read: IOPS=154k, BW=75.2MiB/s '
                                            '(78.8MB/s)(526MiB/7001msec)\n'
                                            '    slat (nsec): min=1673, '
                                            'max=19667, avg=1817.27, '
                                            'stdev=149.65\n'
                                            '    clat (nsec): min=240, '
                                            'max=99747, avg=4449.05, '
                                            'stdev=1090.89\n'
                                            '     lat (usec): min=5, max=101, '
                                            'avg= 6.27, stdev= 1.10\n'
                                            '    clat percentiles (nsec):\n'
                                            '     |  1.00th=[ 4016],  5.00th=[ '
                                            '4192], 10.00th=[ 4256], 20.00th=[ '
                                            '4256],\n'
                                            '     | 30.00th=[ 4256], 40.00th=[ '
                                            '4320], 50.00th=[ 4320], 60.00th=[ '
                                            '4320],\n'
                                            '     | 70.00th=[ 4384], 80.00th=[ '
                                            '4384], 90.00th=[ 4448], 95.00th=[ '
                                            '4512],\n'
                                            '     | 99.00th=[ 9664], '
                                            '99.50th=[10688], 99.90th=[15296], '
                                            '99.95th=[15424],\n'
                                            '     | 99.99th=[57088]\n'
                                            '   bw (  KiB/s): min=76849, '
                                            'max=77209, per=100.00%, '
                                            'avg=77006.79, stdev=107.47, '
                                            'samples=14\n'
                                            '   iops        : min=153699, '
                                            'max=154418, avg=154013.71, '
                                            'stdev=214.80, samples=14\n'
                                            '  lat (nsec)   : 250=0.01%, '
                                            '500=0.01%, 750=0.01%\n'
                                            '  lat (usec)   : 2=0.01%, '
                                            '4=0.96%, 10=98.17%, 20=0.86%, '
                                            '50=0.01%\n'
                                            '  lat (usec)   : 100=0.01%\n'
                                            '  cpu          : usr=16.27%, '
                                            'sys=83.71%, ctx=7, majf=0, '
                                            'minf=0\n'
                                            '  IO depths    : 1=100.0%, '
                                            '2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, >=64=0.0%\n'
                                            '     submit    : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     complete  : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     issued rwts: '
                                            'total=1077656,0,0,0 short=0,0,0,0 '
                                            'dropped=0,0,0,0\n'
                                            '     latency   : target=0, '
                                            'window=0, percentile=100.00%, '
                                            'depth=1\n'
                                            '\n'
                                            'Run status group 0 (all jobs):\n'
                                            '   READ: bw=75.2MiB/s (78.8MB/s), '
                                            '75.2MiB/s-75.2MiB/s '
                                            '(78.8MB/s-78.8MB/s), io=526MiB '
                                            '(552MB), run=7001-7001msec\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_48.output'),
                                  'state': {'begin': 1666782276.5220842,
                                            'cmd': 'cat '
                                                   '/tmp/fio-output_512_1_io_uring-bdev_xnvme_0.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.8515095710754395,
                                            'end': 1666782277.3735938,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_48.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_48.state')},
                       'cmd_49': {'output': 'rm: cannot remove '
                                            "'/tmp/fio-output_512_1_io_uring-bdev_xnvme_1.txt': "
                                            'No such file or directory\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_49.output'),
                                  'state': {'begin': 1666782279.6967657,
                                            'cmd': 'rm '
                                                   '/tmp/fio-output_512_1_io_uring-bdev_xnvme_1.txt',
                                            'cwd': 'None',
                                            'elapsed': 1.0451750755310059,
                                            'end': 1666782280.7419407,
                                            'err': 1,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_49.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_49.state')},
                       'cmd_50': {'output': 'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_50.output'),
                                  'state': {'begin': 1666782280.7438326,
                                            'cmd': '/opt/fio/bin/fio '
                                                   '--name="io_uring-bdev_xnvme" '
                                                   '--rw="randread" '
                                                   '--size="12G" --iodepth="1" '
                                                   '--bs="512" --direct="1" '
                                                   '--thread="1" '
                                                   '--time_based="1" '
                                                   '--runtime="7" '
                                                   '--ramp_time="3" '
                                                   '--norandommap="1" '
                                                   '--allow_file_create="0" '
                                                   '--ioengine="spdk_bdev" '
                                                   '--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" '
                                                   '--filename="Nvme0n1" '
                                                   '--output-format="normal,json" '
                                                   '--output="/tmp/fio-output_512_1_io_uring-bdev_xnvme_1.txt" '
                                                   '--cpus_allowed="1"',
                                            'cwd': 'None',
                                            'elapsed': 11.476299524307251,
                                            'end': 1666782292.220132,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_50.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_50.state')},
                       'cmd_51': {'output': 'io_uring-bdev_xnvme: (g=0): '
                                            'rw=randread, bs=(R) 512B-512B, '
                                            '(W) 512B-512B, (T) 512B-512B, '
                                            'ioengine=spdk_bdev, iodepth=1\n'
                                            'fio-3.32-dirty\n'
                                            'Starting 1 thread\n'
                                            '{\n'
                                            '  "fio version" : '
                                            '"fio-3.32-dirty",\n'
                                            '  "timestamp" : 1666789382,\n'
                                            '  "timestamp_ms" : '
                                            '1666789382237,\n'
                                            '  "time" : "Wed Oct 26 15:03:02 '
                                            '2022",\n'
                                            '  "jobs" : [\n'
                                            '    {\n'
                                            '      "jobname" : '
                                            '"io_uring-bdev_xnvme",\n'
                                            '      "groupid" : 0,\n'
                                            '      "error" : 0,\n'
                                            '      "eta" : 0,\n'
                                            '      "elapsed" : 11,\n'
                                            '      "job options" : {\n'
                                            '        "name" : '
                                            '"io_uring-bdev_xnvme",\n'
                                            '        "rw" : "randread",\n'
                                            '        "size" : "12G",\n'
                                            '        "iodepth" : "1",\n'
                                            '        "bs" : "512",\n'
                                            '        "direct" : "1",\n'
                                            '        "thread" : "1",\n'
                                            '        "time_based" : "1",\n'
                                            '        "runtime" : "7",\n'
                                            '        "ramp_time" : "3",\n'
                                            '        "norandommap" : "1",\n'
                                            '        "allow_file_create" : '
                                            '"0",\n'
                                            '        "ioengine" : '
                                            '"spdk_bdev",\n'
                                            '        "spdk_conf" : '
                                            '"/tmp/spdk_bdev_xnvme_io_uring.conf",\n'
                                            '        "filename" : "Nvme0n1",\n'
                                            '        "cpus_allowed" : "1"\n'
                                            '      },\n'
                                            '      "read" : {\n'
                                            '        "io_bytes" : 552491520,\n'
                                            '        "io_kbytes" : 539542,\n'
                                            '        "bw_bytes" : 78916086,\n'
                                            '        "bw" : 77066,\n'
                                            '        "iops" : 154132.981003,\n'
                                            '        "runtime" : 7001,\n'
                                            '        "total_ios" : 1079085,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 1643,\n'
                                            '          "max" : 21780,\n'
                                            '          "mean" : 1806.536104,\n'
                                            '          "stddev" : 149.212612,\n'
                                            '          "N" : 1079085\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 260,\n'
                                            '          "max" : 101110,\n'
                                            '          "mean" : 4453.978682,\n'
                                            '          "stddev" : '
                                            '1094.494781,\n'
                                            '          "N" : 1079085,\n'
                                            '          "percentile" : {\n'
                                            '            "1.000000" : 4016,\n'
                                            '            "5.000000" : 4192,\n'
                                            '            "10.000000" : 4256,\n'
                                            '            "20.000000" : 4256,\n'
                                            '            "30.000000" : 4256,\n'
                                            '            "40.000000" : 4320,\n'
                                            '            "50.000000" : 4320,\n'
                                            '            "60.000000" : 4320,\n'
                                            '            "70.000000" : 4384,\n'
                                            '            "80.000000" : 4384,\n'
                                            '            "90.000000" : 4448,\n'
                                            '            "95.000000" : 4576,\n'
                                            '            "99.000000" : 9664,\n'
                                            '            "99.500000" : 10688,\n'
                                            '            "99.900000" : 15296,\n'
                                            '            "99.950000" : 15552,\n'
                                            '            "99.990000" : 57088\n'
                                            '          }\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 5510,\n'
                                            '          "max" : 102933,\n'
                                            '          "mean" : 6260.514786,\n'
                                            '          "stddev" : '
                                            '1106.115216,\n'
                                            '          "N" : 1079085\n'
                                            '        },\n'
                                            '        "bw_min" : 76941,\n'
                                            '        "bw_max" : 77411,\n'
                                            '        "bw_agg" : 100.000000,\n'
                                            '        "bw_mean" : '
                                            '77108.928571,\n'
                                            '        "bw_dev" : 145.888506,\n'
                                            '        "bw_samples" : 14,\n'
                                            '        "iops_min" : 153883,\n'
                                            '        "iops_max" : 154822,\n'
                                            '        "iops_mean" : '
                                            '154218.000000,\n'
                                            '        "iops_stddev" : '
                                            '291.839524,\n'
                                            '        "iops_samples" : 14\n'
                                            '      },\n'
                                            '      "write" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "trim" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "sync" : {\n'
                                            '        "total_ios" : 0,\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        }\n'
                                            '      },\n'
                                            '      "job_runtime" : 7000,\n'
                                            '      "usr_cpu" : 17.014286,\n'
                                            '      "sys_cpu" : 82.971429,\n'
                                            '      "ctx" : 10,\n'
                                            '      "majf" : 0,\n'
                                            '      "minf" : 0,\n'
                                            '      "iodepth_level" : {\n'
                                            '        "1" : 100.000000,\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_submit" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_complete" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ns" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.010000,\n'
                                            '        "750" : 0.010000,\n'
                                            '        "1000" : 0.010000\n'
                                            '      },\n'
                                            '      "latency_us" : {\n'
                                            '        "2" : 0.010000,\n'
                                            '        "4" : 0.922634,\n'
                                            '        "10" : 98.195230,\n'
                                            '        "20" : 0.862768,\n'
                                            '        "50" : 0.010000,\n'
                                            '        "100" : 0.012140,\n'
                                            '        "250" : 0.010000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ms" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000,\n'
                                            '        "2000" : 0.000000,\n'
                                            '        ">=2000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_depth" : 1,\n'
                                            '      "latency_target" : 0,\n'
                                            '      "latency_percentile" : '
                                            '100.000000,\n'
                                            '      "latency_window" : 0\n'
                                            '    }\n'
                                            '  ]\n'
                                            '}\n'
                                            '\n'
                                            'io_uring-bdev_xnvme: (groupid=0, '
                                            'jobs=1): err= 0: pid=5770: Wed '
                                            'Oct 26 15:03:02 2022\n'
                                            '  read: IOPS=154k, BW=75.3MiB/s '
                                            '(78.9MB/s)(527MiB/7001msec)\n'
                                            '    slat (nsec): min=1643, '
                                            'max=21780, avg=1806.54, '
                                            'stdev=149.21\n'
                                            '    clat (nsec): min=260, '
                                            'max=101110, avg=4453.98, '
                                            'stdev=1094.49\n'
                                            '     lat (usec): min=5, max=102, '
                                            'avg= 6.26, stdev= 1.11\n'
                                            '    clat percentiles (nsec):\n'
                                            '     |  1.00th=[ 4016],  5.00th=[ '
                                            '4192], 10.00th=[ 4256], 20.00th=[ '
                                            '4256],\n'
                                            '     | 30.00th=[ 4256], 40.00th=[ '
                                            '4320], 50.00th=[ 4320], 60.00th=[ '
                                            '4320],\n'
                                            '     | 70.00th=[ 4384], 80.00th=[ '
                                            '4384], 90.00th=[ 4448], 95.00th=[ '
                                            '4576],\n'
                                            '     | 99.00th=[ 9664], '
                                            '99.50th=[10688], 99.90th=[15296], '
                                            '99.95th=[15552],\n'
                                            '     | 99.99th=[57088]\n'
                                            '   bw (  KiB/s): min=76941, '
                                            'max=77411, per=100.00%, '
                                            'avg=77108.93, stdev=145.89, '
                                            'samples=14\n'
                                            '   iops        : min=153883, '
                                            'max=154822, avg=154218.00, '
                                            'stdev=291.84, samples=14\n'
                                            '  lat (nsec)   : 500=0.01%, '
                                            '750=0.01%, 1000=0.01%\n'
                                            '  lat (usec)   : 2=0.01%, '
                                            '4=0.92%, 10=98.20%, 20=0.86%, '
                                            '50=0.01%\n'
                                            '  lat (usec)   : 100=0.01%, '
                                            '250=0.01%\n'
                                            '  cpu          : usr=17.01%, '
                                            'sys=82.97%, ctx=10, majf=0, '
                                            'minf=0\n'
                                            '  IO depths    : 1=100.0%, '
                                            '2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, >=64=0.0%\n'
                                            '     submit    : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     complete  : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     issued rwts: '
                                            'total=1079085,0,0,0 short=0,0,0,0 '
                                            'dropped=0,0,0,0\n'
                                            '     latency   : target=0, '
                                            'window=0, percentile=100.00%, '
                                            'depth=1\n'
                                            '\n'
                                            'Run status group 0 (all jobs):\n'
                                            '   READ: bw=75.3MiB/s (78.9MB/s), '
                                            '75.3MiB/s-75.3MiB/s '
                                            '(78.9MB/s-78.9MB/s), io=527MiB '
                                            '(552MB), run=7001-7001msec\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_51.output'),
                                  'state': {'begin': 1666782292.2229006,
                                            'cmd': 'cat '
                                                   '/tmp/fio-output_512_1_io_uring-bdev_xnvme_1.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.8048901557922363,
                                            'end': 1666782293.0277908,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_51.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_51.state')},
                       'cmd_52': {'output': 'rm: cannot remove '
                                            "'/tmp/fio-output_512_1_io_uring-bdev_xnvme_2.txt': "
                                            'No such file or directory\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_52.output'),
                                  'state': {'begin': 1666782295.1316152,
                                            'cmd': 'rm '
                                                   '/tmp/fio-output_512_1_io_uring-bdev_xnvme_2.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.8026421070098877,
                                            'end': 1666782295.9342573,
                                            'err': 1,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_52.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_52.state')},
                       'cmd_53': {'output': 'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_53.output'),
                                  'state': {'begin': 1666782295.9411457,
                                            'cmd': '/opt/fio/bin/fio '
                                                   '--name="io_uring-bdev_xnvme" '
                                                   '--rw="randread" '
                                                   '--size="12G" --iodepth="1" '
                                                   '--bs="512" --direct="1" '
                                                   '--thread="1" '
                                                   '--time_based="1" '
                                                   '--runtime="7" '
                                                   '--ramp_time="3" '
                                                   '--norandommap="1" '
                                                   '--allow_file_create="0" '
                                                   '--ioengine="spdk_bdev" '
                                                   '--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" '
                                                   '--filename="Nvme0n1" '
                                                   '--output-format="normal,json" '
                                                   '--output="/tmp/fio-output_512_1_io_uring-bdev_xnvme_2.txt" '
                                                   '--cpus_allowed="1"',
                                            'cwd': 'None',
                                            'elapsed': 11.441778421401978,
                                            'end': 1666782307.382924,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_53.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_53.state')},
                       'cmd_54': {'output': 'io_uring-bdev_xnvme: (g=0): '
                                            'rw=randread, bs=(R) 512B-512B, '
                                            '(W) 512B-512B, (T) 512B-512B, '
                                            'ioengine=spdk_bdev, iodepth=1\n'
                                            'fio-3.32-dirty\n'
                                            'Starting 1 thread\n'
                                            '{\n'
                                            '  "fio version" : '
                                            '"fio-3.32-dirty",\n'
                                            '  "timestamp" : 1666789397,\n'
                                            '  "timestamp_ms" : '
                                            '1666789397412,\n'
                                            '  "time" : "Wed Oct 26 15:03:17 '
                                            '2022",\n'
                                            '  "jobs" : [\n'
                                            '    {\n'
                                            '      "jobname" : '
                                            '"io_uring-bdev_xnvme",\n'
                                            '      "groupid" : 0,\n'
                                            '      "error" : 0,\n'
                                            '      "eta" : 0,\n'
                                            '      "elapsed" : 11,\n'
                                            '      "job options" : {\n'
                                            '        "name" : '
                                            '"io_uring-bdev_xnvme",\n'
                                            '        "rw" : "randread",\n'
                                            '        "size" : "12G",\n'
                                            '        "iodepth" : "1",\n'
                                            '        "bs" : "512",\n'
                                            '        "direct" : "1",\n'
                                            '        "thread" : "1",\n'
                                            '        "time_based" : "1",\n'
                                            '        "runtime" : "7",\n'
                                            '        "ramp_time" : "3",\n'
                                            '        "norandommap" : "1",\n'
                                            '        "allow_file_create" : '
                                            '"0",\n'
                                            '        "ioengine" : '
                                            '"spdk_bdev",\n'
                                            '        "spdk_conf" : '
                                            '"/tmp/spdk_bdev_xnvme_io_uring.conf",\n'
                                            '        "filename" : "Nvme0n1",\n'
                                            '        "cpus_allowed" : "1"\n'
                                            '      },\n'
                                            '      "read" : {\n'
                                            '        "io_bytes" : 549262848,\n'
                                            '        "io_kbytes" : 536389,\n'
                                            '        "bw_bytes" : 78454913,\n'
                                            '        "bw" : 76616,\n'
                                            '        "iops" : 153232.252535,\n'
                                            '        "runtime" : 7001,\n'
                                            '        "total_ios" : 1072779,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 1663,\n'
                                            '          "max" : 17694,\n'
                                            '          "mean" : 1826.517569,\n'
                                            '          "stddev" : 127.585417,\n'
                                            '          "N" : 1072779\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 250,\n'
                                            '          "max" : 114004,\n'
                                            '          "mean" : 4474.261006,\n'
                                            '          "stddev" : '
                                            '1097.446190,\n'
                                            '          "N" : 1072779,\n'
                                            '          "percentile" : {\n'
                                            '            "1.000000" : 4016,\n'
                                            '            "5.000000" : 4256,\n'
                                            '            "10.000000" : 4256,\n'
                                            '            "20.000000" : 4256,\n'
                                            '            "30.000000" : 4320,\n'
                                            '            "40.000000" : 4320,\n'
                                            '            "50.000000" : 4320,\n'
                                            '            "60.000000" : 4384,\n'
                                            '            "70.000000" : 4384,\n'
                                            '            "80.000000" : 4448,\n'
                                            '            "90.000000" : 4448,\n'
                                            '            "95.000000" : 4576,\n'
                                            '            "99.000000" : 9792,\n'
                                            '            "99.500000" : 10688,\n'
                                            '            "99.900000" : 15296,\n'
                                            '            "99.950000" : 15424,\n'
                                            '            "99.990000" : 57088\n'
                                            '          }\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 5540,\n'
                                            '          "max" : 115927,\n'
                                            '          "mean" : 6300.778575,\n'
                                            '          "stddev" : '
                                            '1106.768573,\n'
                                            '          "N" : 1072779\n'
                                            '        },\n'
                                            '        "bw_min" : 76414,\n'
                                            '        "bw_max" : 76744,\n'
                                            '        "bw_agg" : 100.000000,\n'
                                            '        "bw_mean" : '
                                            '76630.071429,\n'
                                            '        "bw_dev" : 96.866652,\n'
                                            '        "bw_samples" : 14,\n'
                                            '        "iops_min" : 152828,\n'
                                            '        "iops_max" : 153488,\n'
                                            '        "iops_mean" : '
                                            '153260.357143,\n'
                                            '        "iops_stddev" : '
                                            '193.638683,\n'
                                            '        "iops_samples" : 14\n'
                                            '      },\n'
                                            '      "write" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "trim" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "sync" : {\n'
                                            '        "total_ios" : 0,\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        }\n'
                                            '      },\n'
                                            '      "job_runtime" : 7000,\n'
                                            '      "usr_cpu" : 21.700000,\n'
                                            '      "sys_cpu" : 78.285714,\n'
                                            '      "ctx" : 9,\n'
                                            '      "majf" : 0,\n'
                                            '      "minf" : 0,\n'
                                            '      "iodepth_level" : {\n'
                                            '        "1" : 100.000000,\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_submit" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_complete" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ns" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.010000,\n'
                                            '        "750" : 0.010000,\n'
                                            '        "1000" : 0.010000\n'
                                            '      },\n'
                                            '      "latency_us" : {\n'
                                            '        "2" : 0.010000,\n'
                                            '        "4" : 0.863552,\n'
                                            '        "10" : 98.266745,\n'
                                            '        "20" : 0.853577,\n'
                                            '        "50" : 0.010000,\n'
                                            '        "100" : 0.011466,\n'
                                            '        "250" : 0.010000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ms" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000,\n'
                                            '        "2000" : 0.000000,\n'
                                            '        ">=2000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_depth" : 1,\n'
                                            '      "latency_target" : 0,\n'
                                            '      "latency_percentile" : '
                                            '100.000000,\n'
                                            '      "latency_window" : 0\n'
                                            '    }\n'
                                            '  ]\n'
                                            '}\n'
                                            '\n'
                                            'io_uring-bdev_xnvme: (groupid=0, '
                                            'jobs=1): err= 0: pid=5827: Wed '
                                            'Oct 26 15:03:17 2022\n'
                                            '  read: IOPS=153k, BW=74.8MiB/s '
                                            '(78.5MB/s)(524MiB/7001msec)\n'
                                            '    slat (nsec): min=1663, '
                                            'max=17694, avg=1826.52, '
                                            'stdev=127.59\n'
                                            '    clat (nsec): min=250, '
                                            'max=114004, avg=4474.26, '
                                            'stdev=1097.45\n'
                                            '     lat (usec): min=5, max=115, '
                                            'avg= 6.30, stdev= 1.11\n'
                                            '    clat percentiles (nsec):\n'
                                            '     |  1.00th=[ 4016],  5.00th=[ '
                                            '4256], 10.00th=[ 4256], 20.00th=[ '
                                            '4256],\n'
                                            '     | 30.00th=[ 4320], 40.00th=[ '
                                            '4320], 50.00th=[ 4320], 60.00th=[ '
                                            '4384],\n'
                                            '     | 70.00th=[ 4384], 80.00th=[ '
                                            '4448], 90.00th=[ 4448], 95.00th=[ '
                                            '4576],\n'
                                            '     | 99.00th=[ 9792], '
                                            '99.50th=[10688], 99.90th=[15296], '
                                            '99.95th=[15424],\n'
                                            '     | 99.99th=[57088]\n'
                                            '   bw (  KiB/s): min=76414, '
                                            'max=76744, per=100.00%, '
                                            'avg=76630.07, stdev=96.87, '
                                            'samples=14\n'
                                            '   iops        : min=152828, '
                                            'max=153488, avg=153260.36, '
                                            'stdev=193.64, samples=14\n'
                                            '  lat (nsec)   : 500=0.01%, '
                                            '750=0.01%, 1000=0.01%\n'
                                            '  lat (usec)   : 2=0.01%, '
                                            '4=0.86%, 10=98.27%, 20=0.85%, '
                                            '50=0.01%\n'
                                            '  lat (usec)   : 100=0.01%, '
                                            '250=0.01%\n'
                                            '  cpu          : usr=21.70%, '
                                            'sys=78.29%, ctx=9, majf=0, '
                                            'minf=0\n'
                                            '  IO depths    : 1=100.0%, '
                                            '2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, >=64=0.0%\n'
                                            '     submit    : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     complete  : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     issued rwts: '
                                            'total=1072779,0,0,0 short=0,0,0,0 '
                                            'dropped=0,0,0,0\n'
                                            '     latency   : target=0, '
                                            'window=0, percentile=100.00%, '
                                            'depth=1\n'
                                            '\n'
                                            'Run status group 0 (all jobs):\n'
                                            '   READ: bw=74.8MiB/s (78.5MB/s), '
                                            '74.8MiB/s-74.8MiB/s '
                                            '(78.5MB/s-78.5MB/s), io=524MiB '
                                            '(549MB), run=7001-7001msec\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_54.output'),
                                  'state': {'begin': 1666782307.3854988,
                                            'cmd': 'cat '
                                                   '/tmp/fio-output_512_1_io_uring-bdev_xnvme_2.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.8249804973602295,
                                            'end': 1666782308.2104793,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_54.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_54.state')},
                       'cmd_55': {'output': 'rm: cannot remove '
                                            "'/tmp/fio-output_512_1_io_uring-xnvme_0.txt': "
                                            'No such file or directory\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_55.output'),
                                  'state': {'begin': 1666782309.2899835,
                                            'cmd': 'rm '
                                                   '/tmp/fio-output_512_1_io_uring-xnvme_0.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.8343324661254883,
                                            'end': 1666782310.124316,
                                            'err': 1,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_55.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_55.state')},
                       'cmd_56': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_56.output'),
                                  'state': {'begin': 1666782310.1264532,
                                            'cmd': '/opt/fio/bin/fio '
                                                   '--name="io_uring-xnvme" '
                                                   '--rw="randread" '
                                                   '--size="12G" --iodepth="1" '
                                                   '--bs="512" --direct="1" '
                                                   '--thread="1" '
                                                   '--time_based="1" '
                                                   '--runtime="7" '
                                                   '--ramp_time="3" '
                                                   '--norandommap="1" '
                                                   '--allow_file_create="0" '
                                                   '--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
                                                   '--xnvme_be="linux" '
                                                   '--xnvme_async="io_uring" '
                                                   '--xnvme_sync="nvme" '
                                                   '--xnvme_admin="nvme" '
                                                   '--xnvme_dev_nsid="1" '
                                                   '--filename="/dev/nvme0n1" '
                                                   '--output-format="normal,json" '
                                                   '--output="/tmp/fio-output_512_1_io_uring-xnvme_0.txt" '
                                                   '--cpus_allowed="1"',
                                            'cwd': 'None',
                                            'elapsed': 11.075448036193848,
                                            'end': 1666782321.2019012,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_56.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_56.state')},
                       'cmd_57': {'output': 'io_uring-xnvme: (g=0): '
                                            'rw=randread, bs=(R) 512B-512B, '
                                            '(W) 512B-512B, (T) 512B-512B, '
                                            'ioengine=xnvme, iodepth=1\n'
                                            'fio-3.32-dirty\n'
                                            'Starting 1 thread\n'
                                            '{\n'
                                            '  "fio version" : '
                                            '"fio-3.32-dirty",\n'
                                            '  "timestamp" : 1666789411,\n'
                                            '  "timestamp_ms" : '
                                            '1666789411269,\n'
                                            '  "time" : "Wed Oct 26 15:03:31 '
                                            '2022",\n'
                                            '  "jobs" : [\n'
                                            '    {\n'
                                            '      "jobname" : '
                                            '"io_uring-xnvme",\n'
                                            '      "groupid" : 0,\n'
                                            '      "error" : 0,\n'
                                            '      "eta" : 0,\n'
                                            '      "elapsed" : 11,\n'
                                            '      "job options" : {\n'
                                            '        "name" : '
                                            '"io_uring-xnvme",\n'
                                            '        "rw" : "randread",\n'
                                            '        "size" : "12G",\n'
                                            '        "iodepth" : "1",\n'
                                            '        "bs" : "512",\n'
                                            '        "direct" : "1",\n'
                                            '        "thread" : "1",\n'
                                            '        "time_based" : "1",\n'
                                            '        "runtime" : "7",\n'
                                            '        "ramp_time" : "3",\n'
                                            '        "norandommap" : "1",\n'
                                            '        "allow_file_create" : '
                                            '"0",\n'
                                            '        "ioengine" : '
                                            '"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
                                            '        "xnvme_be" : "linux",\n'
                                            '        "xnvme_async" : '
                                            '"io_uring",\n'
                                            '        "xnvme_sync" : "nvme",\n'
                                            '        "xnvme_admin" : "nvme",\n'
                                            '        "xnvme_dev_nsid" : "1",\n'
                                            '        "filename" : '
                                            '"/dev/nvme0n1",\n'
                                            '        "cpus_allowed" : "1"\n'
                                            '      },\n'
                                            '      "read" : {\n'
                                            '        "io_bytes" : 390937600,\n'
                                            '        "io_kbytes" : 381775,\n'
                                            '        "bw_bytes" : 55840251,\n'
                                            '        "bw" : 54531,\n'
                                            '        "iops" : 109062.991001,\n'
                                            '        "runtime" : 7001,\n'
                                            '        "total_ios" : 763550,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 1503,\n'
                                            '          "max" : 14387,\n'
                                            '          "mean" : 1667.139119,\n'
                                            '          "stddev" : 81.310916,\n'
                                            '          "N" : 763550\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 230,\n'
                                            '          "max" : 118001,\n'
                                            '          "mean" : 7204.880431,\n'
                                            '          "stddev" : 960.561063,\n'
                                            '          "N" : 763550,\n'
                                            '          "percentile" : {\n'
                                            '            "1.000000" : 6752,\n'
                                            '            "5.000000" : 6944,\n'
                                            '            "10.000000" : 6944,\n'
                                            '            "20.000000" : 7008,\n'
                                            '            "30.000000" : 7008,\n'
                                            '            "40.000000" : 7072,\n'
                                            '            "50.000000" : 7136,\n'
                                            '            "60.000000" : 7136,\n'
                                            '            "70.000000" : 7200,\n'
                                            '            "80.000000" : 7200,\n'
                                            '            "90.000000" : 7264,\n'
                                            '            "95.000000" : 7392,\n'
                                            '            "99.000000" : 10176,\n'
                                            '            "99.500000" : 13504,\n'
                                            '            "99.900000" : 15552,\n'
                                            '            "99.950000" : 15808,\n'
                                            '            "99.990000" : 57600\n'
                                            '          }\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 7204,\n'
                                            '          "max" : 119654,\n'
                                            '          "mean" : 8872.019549,\n'
                                            '          "stddev" : 965.822097,\n'
                                            '          "N" : 763550\n'
                                            '        },\n'
                                            '        "bw_min" : 54161,\n'
                                            '        "bw_max" : 54881,\n'
                                            '        "bw_agg" : 100.000000,\n'
                                            '        "bw_mean" : '
                                            '54540.928571,\n'
                                            '        "bw_dev" : 272.108424,\n'
                                            '        "bw_samples" : 14,\n'
                                            '        "iops_min" : 108322,\n'
                                            '        "iops_max" : 109762,\n'
                                            '        "iops_mean" : '
                                            '109081.928571,\n'
                                            '        "iops_stddev" : '
                                            '544.151628,\n'
                                            '        "iops_samples" : 14\n'
                                            '      },\n'
                                            '      "write" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "trim" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "sync" : {\n'
                                            '        "total_ios" : 0,\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        }\n'
                                            '      },\n'
                                            '      "job_runtime" : 7000,\n'
                                            '      "usr_cpu" : 8.357143,\n'
                                            '      "sys_cpu" : 36.128571,\n'
                                            '      "ctx" : 763553,\n'
                                            '      "majf" : 0,\n'
                                            '      "minf" : 0,\n'
                                            '      "iodepth_level" : {\n'
                                            '        "1" : 100.000000,\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_submit" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_complete" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ns" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.010000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_us" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.010000,\n'
                                            '        "10" : 98.860847,\n'
                                            '        "20" : 1.121865,\n'
                                            '        "50" : 0.010000,\n'
                                            '        "100" : 0.013228,\n'
                                            '        "250" : 0.010000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ms" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000,\n'
                                            '        "2000" : 0.000000,\n'
                                            '        ">=2000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_depth" : 1,\n'
                                            '      "latency_target" : 0,\n'
                                            '      "latency_percentile" : '
                                            '100.000000,\n'
                                            '      "latency_window" : 0\n'
                                            '    }\n'
                                            '  ]\n'
                                            '}\n'
                                            '\n'
                                            'io_uring-xnvme: (groupid=0, '
                                            'jobs=1): err= 0: pid=5874: Wed '
                                            'Oct 26 15:03:31 2022\n'
                                            '  read: IOPS=109k, BW=53.3MiB/s '
                                            '(55.8MB/s)(373MiB/7001msec)\n'
                                            '    slat (nsec): min=1503, '
                                            'max=14387, avg=1667.14, '
                                            'stdev=81.31\n'
                                            '    clat (nsec): min=230, '
                                            'max=118001, avg=7204.88, '
                                            'stdev=960.56\n'
                                            '     lat (nsec): min=7204, '
                                            'max=119654, avg=8872.02, '
                                            'stdev=965.82\n'
                                            '    clat percentiles (nsec):\n'
                                            '     |  1.00th=[ 6752],  5.00th=[ '
                                            '6944], 10.00th=[ 6944], 20.00th=[ '
                                            '7008],\n'
                                            '     | 30.00th=[ 7008], 40.00th=[ '
                                            '7072], 50.00th=[ 7136], 60.00th=[ '
                                            '7136],\n'
                                            '     | 70.00th=[ 7200], 80.00th=[ '
                                            '7200], 90.00th=[ 7264], 95.00th=[ '
                                            '7392],\n'
                                            '     | 99.00th=[10176], '
                                            '99.50th=[13504], 99.90th=[15552], '
                                            '99.95th=[15808],\n'
                                            '     | 99.99th=[57600]\n'
                                            '   bw (  KiB/s): min=54161, '
                                            'max=54881, per=100.00%, '
                                            'avg=54540.93, stdev=272.11, '
                                            'samples=14\n'
                                            '   iops        : min=108322, '
                                            'max=109762, avg=109081.93, '
                                            'stdev=544.15, samples=14\n'
                                            '  lat (nsec)   : 250=0.01%\n'
                                            '  lat (usec)   : 4=0.01%, '
                                            '10=98.86%, 20=1.12%, 50=0.01%, '
                                            '100=0.01%\n'
                                            '  lat (usec)   : 250=0.01%\n'
                                            '  cpu          : usr=8.36%, '
                                            'sys=36.13%, ctx=763553, majf=0, '
                                            'minf=0\n'
                                            '  IO depths    : 1=100.0%, '
                                            '2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, >=64=0.0%\n'
                                            '     submit    : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     complete  : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     issued rwts: '
                                            'total=763550,0,0,0 short=0,0,0,0 '
                                            'dropped=0,0,0,0\n'
                                            '     latency   : target=0, '
                                            'window=0, percentile=100.00%, '
                                            'depth=1\n'
                                            '\n'
                                            'Run status group 0 (all jobs):\n'
                                            '   READ: bw=53.3MiB/s (55.8MB/s), '
                                            '53.3MiB/s-53.3MiB/s '
                                            '(55.8MB/s-55.8MB/s), io=373MiB '
                                            '(391MB), run=7001-7001msec\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_57.output'),
                                  'state': {'begin': 1666782321.2053792,
                                            'cmd': 'cat '
                                                   '/tmp/fio-output_512_1_io_uring-xnvme_0.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.8669404983520508,
                                            'end': 1666782322.0723197,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_57.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_57.state')},
                       'cmd_58': {'output': 'rm: cannot remove '
                                            "'/tmp/fio-output_512_1_io_uring-xnvme_1.txt': "
                                            'No such file or directory\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_58.output'),
                                  'state': {'begin': 1666782323.139761,
                                            'cmd': 'rm '
                                                   '/tmp/fio-output_512_1_io_uring-xnvme_1.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.8553993701934814,
                                            'end': 1666782323.9951603,
                                            'err': 1,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_58.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_58.state')},
                       'cmd_59': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_59.output'),
                                  'state': {'begin': 1666782323.9972491,
                                            'cmd': '/opt/fio/bin/fio '
                                                   '--name="io_uring-xnvme" '
                                                   '--rw="randread" '
                                                   '--size="12G" --iodepth="1" '
                                                   '--bs="512" --direct="1" '
                                                   '--thread="1" '
                                                   '--time_based="1" '
                                                   '--runtime="7" '
                                                   '--ramp_time="3" '
                                                   '--norandommap="1" '
                                                   '--allow_file_create="0" '
                                                   '--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
                                                   '--xnvme_be="linux" '
                                                   '--xnvme_async="io_uring" '
                                                   '--xnvme_sync="nvme" '
                                                   '--xnvme_admin="nvme" '
                                                   '--xnvme_dev_nsid="1" '
                                                   '--filename="/dev/nvme0n1" '
                                                   '--output-format="normal,json" '
                                                   '--output="/tmp/fio-output_512_1_io_uring-xnvme_1.txt" '
                                                   '--cpus_allowed="1"',
                                            'cwd': 'None',
                                            'elapsed': 11.102566242218018,
                                            'end': 1666782335.0998154,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_59.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_59.state')},
                       'cmd_60': {'output': 'io_uring-xnvme: (g=0): '
                                            'rw=randread, bs=(R) 512B-512B, '
                                            '(W) 512B-512B, (T) 512B-512B, '
                                            'ioengine=xnvme, iodepth=1\n'
                                            'fio-3.32-dirty\n'
                                            'Starting 1 thread\n'
                                            '{\n'
                                            '  "fio version" : '
                                            '"fio-3.32-dirty",\n'
                                            '  "timestamp" : 1666789425,\n'
                                            '  "timestamp_ms" : '
                                            '1666789425191,\n'
                                            '  "time" : "Wed Oct 26 15:03:45 '
                                            '2022",\n'
                                            '  "jobs" : [\n'
                                            '    {\n'
                                            '      "jobname" : '
                                            '"io_uring-xnvme",\n'
                                            '      "groupid" : 0,\n'
                                            '      "error" : 0,\n'
                                            '      "eta" : 0,\n'
                                            '      "elapsed" : 11,\n'
                                            '      "job options" : {\n'
                                            '        "name" : '
                                            '"io_uring-xnvme",\n'
                                            '        "rw" : "randread",\n'
                                            '        "size" : "12G",\n'
                                            '        "iodepth" : "1",\n'
                                            '        "bs" : "512",\n'
                                            '        "direct" : "1",\n'
                                            '        "thread" : "1",\n'
                                            '        "time_based" : "1",\n'
                                            '        "runtime" : "7",\n'
                                            '        "ramp_time" : "3",\n'
                                            '        "norandommap" : "1",\n'
                                            '        "allow_file_create" : '
                                            '"0",\n'
                                            '        "ioengine" : '
                                            '"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
                                            '        "xnvme_be" : "linux",\n'
                                            '        "xnvme_async" : '
                                            '"io_uring",\n'
                                            '        "xnvme_sync" : "nvme",\n'
                                            '        "xnvme_admin" : "nvme",\n'
                                            '        "xnvme_dev_nsid" : "1",\n'
                                            '        "filename" : '
                                            '"/dev/nvme0n1",\n'
                                            '        "cpus_allowed" : "1"\n'
                                            '      },\n'
                                            '      "read" : {\n'
                                            '        "io_bytes" : 390010880,\n'
                                            '        "io_kbytes" : 380870,\n'
                                            '        "bw_bytes" : 55715840,\n'
                                            '        "bw" : 54410,\n'
                                            '        "iops" : 108819.857143,\n'
                                            '        "runtime" : 7000,\n'
                                            '        "total_ios" : 761739,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 1523,\n'
                                            '          "max" : 23895,\n'
                                            '          "mean" : 1676.788178,\n'
                                            '          "stddev" : 77.910712,\n'
                                            '          "N" : 761740\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 211,\n'
                                            '          "max" : 134522,\n'
                                            '          "mean" : 7217.694233,\n'
                                            '          "stddev" : '
                                            '1010.392200,\n'
                                            '          "N" : 761739,\n'
                                            '          "percentile" : {\n'
                                            '            "1.000000" : 6752,\n'
                                            '            "5.000000" : 6944,\n'
                                            '            "10.000000" : 7008,\n'
                                            '            "20.000000" : 7008,\n'
                                            '            "30.000000" : 7008,\n'
                                            '            "40.000000" : 7072,\n'
                                            '            "50.000000" : 7136,\n'
                                            '            "60.000000" : 7136,\n'
                                            '            "70.000000" : 7200,\n'
                                            '            "80.000000" : 7200,\n'
                                            '            "90.000000" : 7328,\n'
                                            '            "95.000000" : 7392,\n'
                                            '            "99.000000" : 10176,\n'
                                            '            "99.500000" : 13632,\n'
                                            '            "99.900000" : 15552,\n'
                                            '            "99.950000" : 15808,\n'
                                            '            "99.990000" : 59648\n'
                                            '          }\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 7003,\n'
                                            '          "max" : 136165,\n'
                                            '          "mean" : 8894.482495,\n'
                                            '          "stddev" : '
                                            '1014.288753,\n'
                                            '          "N" : 761739\n'
                                            '        },\n'
                                            '        "bw_min" : 54271,\n'
                                            '        "bw_max" : 54567,\n'
                                            '        "bw_agg" : 100.000000,\n'
                                            '        "bw_mean" : '
                                            '54411.928571,\n'
                                            '        "bw_dev" : 78.668207,\n'
                                            '        "bw_samples" : 14,\n'
                                            '        "iops_min" : 108542,\n'
                                            '        "iops_max" : 109134,\n'
                                            '        "iops_mean" : '
                                            '108823.928571,\n'
                                            '        "iops_stddev" : '
                                            '157.274119,\n'
                                            '        "iops_samples" : 14\n'
                                            '      },\n'
                                            '      "write" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "trim" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "sync" : {\n'
                                            '        "total_ios" : 0,\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        }\n'
                                            '      },\n'
                                            '      "job_runtime" : 7000,\n'
                                            '      "usr_cpu" : 9.200000,\n'
                                            '      "sys_cpu" : 35.271429,\n'
                                            '      "ctx" : 761738,\n'
                                            '      "majf" : 0,\n'
                                            '      "minf" : 0,\n'
                                            '      "iodepth_level" : {\n'
                                            '        "1" : 100.000000,\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_submit" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_complete" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ns" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.010000,\n'
                                            '        "500" : 0.010000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_us" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.010000,\n'
                                            '        "10" : 98.872448,\n'
                                            '        "20" : 1.107466,\n'
                                            '        "50" : 0.010000,\n'
                                            '        "100" : 0.015228,\n'
                                            '        "250" : 0.010000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ms" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000,\n'
                                            '        "2000" : 0.000000,\n'
                                            '        ">=2000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_depth" : 1,\n'
                                            '      "latency_target" : 0,\n'
                                            '      "latency_percentile" : '
                                            '100.000000,\n'
                                            '      "latency_window" : 0\n'
                                            '    }\n'
                                            '  ]\n'
                                            '}\n'
                                            '\n'
                                            'io_uring-xnvme: (groupid=0, '
                                            'jobs=1): err= 0: pid=5922: Wed '
                                            'Oct 26 15:03:45 2022\n'
                                            '  read: IOPS=109k, BW=53.1MiB/s '
                                            '(55.7MB/s)(372MiB/7000msec)\n'
                                            '    slat (nsec): min=1523, '
                                            'max=23895, avg=1676.79, '
                                            'stdev=77.91\n'
                                            '    clat (nsec): min=211, '
                                            'max=134522, avg=7217.69, '
                                            'stdev=1010.39\n'
                                            '     lat (usec): min=7, max=136, '
                                            'avg= 8.89, stdev= 1.01\n'
                                            '    clat percentiles (nsec):\n'
                                            '     |  1.00th=[ 6752],  5.00th=[ '
                                            '6944], 10.00th=[ 7008], 20.00th=[ '
                                            '7008],\n'
                                            '     | 30.00th=[ 7008], 40.00th=[ '
                                            '7072], 50.00th=[ 7136], 60.00th=[ '
                                            '7136],\n'
                                            '     | 70.00th=[ 7200], 80.00th=[ '
                                            '7200], 90.00th=[ 7328], 95.00th=[ '
                                            '7392],\n'
                                            '     | 99.00th=[10176], '
                                            '99.50th=[13632], 99.90th=[15552], '
                                            '99.95th=[15808],\n'
                                            '     | 99.99th=[59648]\n'
                                            '   bw (  KiB/s): min=54271, '
                                            'max=54567, per=100.00%, '
                                            'avg=54411.93, stdev=78.67, '
                                            'samples=14\n'
                                            '   iops        : min=108542, '
                                            'max=109134, avg=108823.93, '
                                            'stdev=157.27, samples=14\n'
                                            '  lat (nsec)   : 250=0.01%, '
                                            '500=0.01%\n'
                                            '  lat (usec)   : 4=0.01%, '
                                            '10=98.87%, 20=1.11%, 50=0.01%, '
                                            '100=0.02%\n'
                                            '  lat (usec)   : 250=0.01%\n'
                                            '  cpu          : usr=9.20%, '
                                            'sys=35.27%, ctx=761738, majf=0, '
                                            'minf=0\n'
                                            '  IO depths    : 1=100.0%, '
                                            '2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, >=64=0.0%\n'
                                            '     submit    : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     complete  : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     issued rwts: '
                                            'total=761739,0,0,0 short=0,0,0,0 '
                                            'dropped=0,0,0,0\n'
                                            '     latency   : target=0, '
                                            'window=0, percentile=100.00%, '
                                            'depth=1\n'
                                            '\n'
                                            'Run status group 0 (all jobs):\n'
                                            '   READ: bw=53.1MiB/s (55.7MB/s), '
                                            '53.1MiB/s-53.1MiB/s '
                                            '(55.7MB/s-55.7MB/s), io=372MiB '
                                            '(390MB), run=7000-7000msec\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_60.output'),
                                  'state': {'begin': 1666782335.102811,
                                            'cmd': 'cat '
                                                   '/tmp/fio-output_512_1_io_uring-xnvme_1.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.8670730590820312,
                                            'end': 1666782335.9698842,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_60.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_60.state')},
                       'cmd_61': {'output': 'rm: cannot remove '
                                            "'/tmp/fio-output_512_1_io_uring-xnvme_2.txt': "
                                            'No such file or directory\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_61.output'),
                                  'state': {'begin': 1666782337.0434806,
                                            'cmd': 'rm '
                                                   '/tmp/fio-output_512_1_io_uring-xnvme_2.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.8228695392608643,
                                            'end': 1666782337.8663502,
                                            'err': 1,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_61.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_61.state')},
                       'cmd_62': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_62.output'),
                                  'state': {'begin': 1666782337.8692353,
                                            'cmd': '/opt/fio/bin/fio '
                                                   '--name="io_uring-xnvme" '
                                                   '--rw="randread" '
                                                   '--size="12G" --iodepth="1" '
                                                   '--bs="512" --direct="1" '
                                                   '--thread="1" '
                                                   '--time_based="1" '
                                                   '--runtime="7" '
                                                   '--ramp_time="3" '
                                                   '--norandommap="1" '
                                                   '--allow_file_create="0" '
                                                   '--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
                                                   '--xnvme_be="linux" '
                                                   '--xnvme_async="io_uring" '
                                                   '--xnvme_sync="nvme" '
                                                   '--xnvme_admin="nvme" '
                                                   '--xnvme_dev_nsid="1" '
                                                   '--filename="/dev/nvme0n1" '
                                                   '--output-format="normal,json" '
                                                   '--output="/tmp/fio-output_512_1_io_uring-xnvme_2.txt" '
                                                   '--cpus_allowed="1"',
                                            'cwd': 'None',
                                            'elapsed': 11.072468042373657,
                                            'end': 1666782348.9417033,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_62.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_62.state')},
                       'cmd_63': {'output': 'io_uring-xnvme: (g=0): '
                                            'rw=randread, bs=(R) 512B-512B, '
                                            '(W) 512B-512B, (T) 512B-512B, '
                                            'ioengine=xnvme, iodepth=1\n'
                                            'fio-3.32-dirty\n'
                                            'Starting 1 thread\n'
                                            '{\n'
                                            '  "fio version" : '
                                            '"fio-3.32-dirty",\n'
                                            '  "timestamp" : 1666789439,\n'
                                            '  "timestamp_ms" : '
                                            '1666789439029,\n'
                                            '  "time" : "Wed Oct 26 15:03:59 '
                                            '2022",\n'
                                            '  "jobs" : [\n'
                                            '    {\n'
                                            '      "jobname" : '
                                            '"io_uring-xnvme",\n'
                                            '      "groupid" : 0,\n'
                                            '      "error" : 0,\n'
                                            '      "eta" : 0,\n'
                                            '      "elapsed" : 11,\n'
                                            '      "job options" : {\n'
                                            '        "name" : '
                                            '"io_uring-xnvme",\n'
                                            '        "rw" : "randread",\n'
                                            '        "size" : "12G",\n'
                                            '        "iodepth" : "1",\n'
                                            '        "bs" : "512",\n'
                                            '        "direct" : "1",\n'
                                            '        "thread" : "1",\n'
                                            '        "time_based" : "1",\n'
                                            '        "runtime" : "7",\n'
                                            '        "ramp_time" : "3",\n'
                                            '        "norandommap" : "1",\n'
                                            '        "allow_file_create" : '
                                            '"0",\n'
                                            '        "ioengine" : '
                                            '"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
                                            '        "xnvme_be" : "linux",\n'
                                            '        "xnvme_async" : '
                                            '"io_uring",\n'
                                            '        "xnvme_sync" : "nvme",\n'
                                            '        "xnvme_admin" : "nvme",\n'
                                            '        "xnvme_dev_nsid" : "1",\n'
                                            '        "filename" : '
                                            '"/dev/nvme0n1",\n'
                                            '        "cpus_allowed" : "1"\n'
                                            '      },\n'
                                            '      "read" : {\n'
                                            '        "io_bytes" : 390489088,\n'
                                            '        "io_kbytes" : 381337,\n'
                                            '        "bw_bytes" : 55776187,\n'
                                            '        "bw" : 54468,\n'
                                            '        "iops" : 108937.723182,\n'
                                            '        "runtime" : 7001,\n'
                                            '        "total_ios" : 762673,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 1512,\n'
                                            '          "max" : 22973,\n'
                                            '          "mean" : 1663.808307,\n'
                                            '          "stddev" : 81.901888,\n'
                                            '          "N" : 762674\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 2965,\n'
                                            '          "max" : 339156,\n'
                                            '          "mean" : 7211.033603,\n'
                                            '          "stddev" : '
                                            '1048.223194,\n'
                                            '          "N" : 762673,\n'
                                            '          "percentile" : {\n'
                                            '            "1.000000" : 6752,\n'
                                            '            "5.000000" : 6944,\n'
                                            '            "10.000000" : 6944,\n'
                                            '            "20.000000" : 7008,\n'
                                            '            "30.000000" : 7008,\n'
                                            '            "40.000000" : 7072,\n'
                                            '            "50.000000" : 7136,\n'
                                            '            "60.000000" : 7136,\n'
                                            '            "70.000000" : 7200,\n'
                                            '            "80.000000" : 7200,\n'
                                            '            "90.000000" : 7264,\n'
                                            '            "95.000000" : 7392,\n'
                                            '            "99.000000" : 10176,\n'
                                            '            "99.500000" : 13504,\n'
                                            '            "99.900000" : 15552,\n'
                                            '            "99.950000" : 15808,\n'
                                            '            "99.990000" : 57600\n'
                                            '          }\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 7304,\n'
                                            '          "max" : 340799,\n'
                                            '          "mean" : 8874.841846,\n'
                                            '          "stddev" : '
                                            '1052.530746,\n'
                                            '          "N" : 762673\n'
                                            '        },\n'
                                            '        "bw_min" : 54216,\n'
                                            '        "bw_max" : 54781,\n'
                                            '        "bw_agg" : 100.000000,\n'
                                            '        "bw_mean" : '
                                            '54498.571429,\n'
                                            '        "bw_dev" : 162.638681,\n'
                                            '        "bw_samples" : 14,\n'
                                            '        "iops_min" : 108432,\n'
                                            '        "iops_max" : 109562,\n'
                                            '        "iops_mean" : '
                                            '108997.285714,\n'
                                            '        "iops_stddev" : '
                                            '325.114283,\n'
                                            '        "iops_samples" : 14\n'
                                            '      },\n'
                                            '      "write" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "trim" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "sync" : {\n'
                                            '        "total_ios" : 0,\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        }\n'
                                            '      },\n'
                                            '      "job_runtime" : 7000,\n'
                                            '      "usr_cpu" : 10.714286,\n'
                                            '      "sys_cpu" : 33.800000,\n'
                                            '      "ctx" : 762675,\n'
                                            '      "majf" : 0,\n'
                                            '      "minf" : 0,\n'
                                            '      "iodepth_level" : {\n'
                                            '        "1" : 100.000000,\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_submit" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_complete" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ns" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_us" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.010000,\n'
                                            '        "10" : 98.851539,\n'
                                            '        "20" : 1.126957,\n'
                                            '        "50" : 0.010000,\n'
                                            '        "100" : 0.014685,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.010000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ms" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000,\n'
                                            '        "2000" : 0.000000,\n'
                                            '        ">=2000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_depth" : 1,\n'
                                            '      "latency_target" : 0,\n'
                                            '      "latency_percentile" : '
                                            '100.000000,\n'
                                            '      "latency_window" : 0\n'
                                            '    }\n'
                                            '  ]\n'
                                            '}\n'
                                            '\n'
                                            'io_uring-xnvme: (groupid=0, '
                                            'jobs=1): err= 0: pid=5970: Wed '
                                            'Oct 26 15:03:59 2022\n'
                                            '  read: IOPS=109k, BW=53.2MiB/s '
                                            '(55.8MB/s)(372MiB/7001msec)\n'
                                            '    slat (nsec): min=1512, '
                                            'max=22973, avg=1663.81, '
                                            'stdev=81.90\n'
                                            '    clat (usec): min=2, max=339, '
                                            'avg= 7.21, stdev= 1.05\n'
                                            '     lat (usec): min=7, max=340, '
                                            'avg= 8.87, stdev= 1.05\n'
                                            '    clat percentiles (nsec):\n'
                                            '     |  1.00th=[ 6752],  5.00th=[ '
                                            '6944], 10.00th=[ 6944], 20.00th=[ '
                                            '7008],\n'
                                            '     | 30.00th=[ 7008], 40.00th=[ '
                                            '7072], 50.00th=[ 7136], 60.00th=[ '
                                            '7136],\n'
                                            '     | 70.00th=[ 7200], 80.00th=[ '
                                            '7200], 90.00th=[ 7264], 95.00th=[ '
                                            '7392],\n'
                                            '     | 99.00th=[10176], '
                                            '99.50th=[13504], 99.90th=[15552], '
                                            '99.95th=[15808],\n'
                                            '     | 99.99th=[57600]\n'
                                            '   bw (  KiB/s): min=54216, '
                                            'max=54781, per=100.00%, '
                                            'avg=54498.57, stdev=162.64, '
                                            'samples=14\n'
                                            '   iops        : min=108432, '
                                            'max=109562, avg=108997.29, '
                                            'stdev=325.11, samples=14\n'
                                            '  lat (usec)   : 4=0.01%, '
                                            '10=98.85%, 20=1.13%, 50=0.01%, '
                                            '100=0.01%\n'
                                            '  lat (usec)   : 500=0.01%\n'
                                            '  cpu          : usr=10.71%, '
                                            'sys=33.80%, ctx=762675, majf=0, '
                                            'minf=0\n'
                                            '  IO depths    : 1=100.0%, '
                                            '2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, >=64=0.0%\n'
                                            '     submit    : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     complete  : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     issued rwts: '
                                            'total=762673,0,0,0 short=0,0,0,0 '
                                            'dropped=0,0,0,0\n'
                                            '     latency   : target=0, '
                                            'window=0, percentile=100.00%, '
                                            'depth=1\n'
                                            '\n'
                                            'Run status group 0 (all jobs):\n'
                                            '   READ: bw=53.2MiB/s (55.8MB/s), '
                                            '53.2MiB/s-53.2MiB/s '
                                            '(55.8MB/s-55.8MB/s), io=372MiB '
                                            '(390MB), run=7001-7001msec\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_63.output'),
                                  'state': {'begin': 1666782348.9444702,
                                            'cmd': 'cat '
                                                   '/tmp/fio-output_512_1_io_uring-xnvme_2.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.7905919551849365,
                                            'end': 1666782349.7350621,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_63.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_63.state')},
                       'cmd_64': {'output': 'rm: cannot remove '
                                            "'/tmp/fio-output_512_1_libaio-reference_0.txt': "
                                            'No such file or directory\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_64.output'),
                                  'state': {'begin': 1666782350.802364,
                                            'cmd': 'rm '
                                                   '/tmp/fio-output_512_1_libaio-reference_0.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.8477296829223633,
                                            'end': 1666782351.6500938,
                                            'err': 1,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_64.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_64.state')},
                       'cmd_65': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_65.output'),
                                  'state': {'begin': 1666782351.652733,
                                            'cmd': '/opt/fio/bin/fio '
                                                   '--name="libaio-reference" '
                                                   '--rw="randread" '
                                                   '--size="12G" --iodepth="1" '
                                                   '--bs="512" --direct="1" '
                                                   '--thread="1" '
                                                   '--time_based="1" '
                                                   '--runtime="7" '
                                                   '--ramp_time="3" '
                                                   '--norandommap="1" '
                                                   '--allow_file_create="0" '
                                                   '--ioengine="libaio" '
                                                   '--filename="/dev/nvme0n1" '
                                                   '--output-format="normal,json" '
                                                   '--output="/tmp/fio-output_512_1_libaio-reference_0.txt" '
                                                   '--cpus_allowed="1"',
                                            'cwd': 'None',
                                            'elapsed': 11.121761798858643,
                                            'end': 1666782362.774495,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_65.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_65.state')},
                       'cmd_66': {'output': 'libaio-reference: (g=0): '
                                            'rw=randread, bs=(R) 512B-512B, '
                                            '(W) 512B-512B, (T) 512B-512B, '
                                            'ioengine=libaio, iodepth=1\n'
                                            'fio-3.32-dirty\n'
                                            'Starting 1 thread\n'
                                            '{\n'
                                            '  "fio version" : '
                                            '"fio-3.32-dirty",\n'
                                            '  "timestamp" : 1666789452,\n'
                                            '  "timestamp_ms" : '
                                            '1666789452865,\n'
                                            '  "time" : "Wed Oct 26 15:04:12 '
                                            '2022",\n'
                                            '  "jobs" : [\n'
                                            '    {\n'
                                            '      "jobname" : '
                                            '"libaio-reference",\n'
                                            '      "groupid" : 0,\n'
                                            '      "error" : 0,\n'
                                            '      "eta" : 0,\n'
                                            '      "elapsed" : 11,\n'
                                            '      "job options" : {\n'
                                            '        "name" : '
                                            '"libaio-reference",\n'
                                            '        "rw" : "randread",\n'
                                            '        "size" : "12G",\n'
                                            '        "iodepth" : "1",\n'
                                            '        "bs" : "512",\n'
                                            '        "direct" : "1",\n'
                                            '        "thread" : "1",\n'
                                            '        "time_based" : "1",\n'
                                            '        "runtime" : "7",\n'
                                            '        "ramp_time" : "3",\n'
                                            '        "norandommap" : "1",\n'
                                            '        "allow_file_create" : '
                                            '"0",\n'
                                            '        "ioengine" : "libaio",\n'
                                            '        "filename" : '
                                            '"/dev/nvme0n1",\n'
                                            '        "cpus_allowed" : "1"\n'
                                            '      },\n'
                                            '      "read" : {\n'
                                            '        "io_bytes" : 391555584,\n'
                                            '        "io_kbytes" : 382378,\n'
                                            '        "bw_bytes" : 55928522,\n'
                                            '        "bw" : 54617,\n'
                                            '        "iops" : 109235.394944,\n'
                                            '        "runtime" : 7001,\n'
                                            '        "total_ios" : 764757,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 1653,\n'
                                            '          "max" : 6341,\n'
                                            '          "mean" : 1761.576344,\n'
                                            '          "stddev" : 74.909216,\n'
                                            '          "N" : 764757\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 2775,\n'
                                            '          "max" : 94186,\n'
                                            '          "mean" : 7098.314393,\n'
                                            '          "stddev" : 981.590607,\n'
                                            '          "N" : 764757,\n'
                                            '          "percentile" : {\n'
                                            '            "1.000000" : 6624,\n'
                                            '            "5.000000" : 6816,\n'
                                            '            "10.000000" : 6880,\n'
                                            '            "20.000000" : 6880,\n'
                                            '            "30.000000" : 6944,\n'
                                            '            "40.000000" : 6944,\n'
                                            '            "50.000000" : 7008,\n'
                                            '            "60.000000" : 7072,\n'
                                            '            "70.000000" : 7072,\n'
                                            '            "80.000000" : 7072,\n'
                                            '            "90.000000" : 7200,\n'
                                            '            "95.000000" : 7264,\n'
                                            '            "99.000000" : 10176,\n'
                                            '            "99.500000" : 13504,\n'
                                            '            "99.900000" : 15424,\n'
                                            '            "99.950000" : 15680,\n'
                                            '            "99.990000" : 58112\n'
                                            '          }\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 7384,\n'
                                            '          "max" : 95980,\n'
                                            '          "mean" : 8859.890737,\n'
                                            '          "stddev" : 985.744709,\n'
                                            '          "N" : 764757\n'
                                            '        },\n'
                                            '        "bw_min" : 54489,\n'
                                            '        "bw_max" : 54838,\n'
                                            '        "bw_agg" : 100.000000,\n'
                                            '        "bw_mean" : '
                                            '54647.692308,\n'
                                            '        "bw_dev" : 106.501475,\n'
                                            '        "bw_samples" : 13,\n'
                                            '        "iops_min" : 108978,\n'
                                            '        "iops_max" : 109676,\n'
                                            '        "iops_mean" : '
                                            '109295.461538,\n'
                                            '        "iops_stddev" : '
                                            '213.112887,\n'
                                            '        "iops_samples" : 13\n'
                                            '      },\n'
                                            '      "write" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "trim" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "sync" : {\n'
                                            '        "total_ios" : 0,\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        }\n'
                                            '      },\n'
                                            '      "job_runtime" : 7000,\n'
                                            '      "usr_cpu" : 15.628571,\n'
                                            '      "sys_cpu" : 27.585714,\n'
                                            '      "ctx" : 764759,\n'
                                            '      "majf" : 0,\n'
                                            '      "minf" : 1,\n'
                                            '      "iodepth_level" : {\n'
                                            '        "1" : 100.000000,\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_submit" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_complete" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ns" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_us" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.010000,\n'
                                            '        "10" : 98.872583,\n'
                                            '        "20" : 1.104534,\n'
                                            '        "50" : 0.010000,\n'
                                            '        "100" : 0.014253,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ms" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000,\n'
                                            '        "2000" : 0.000000,\n'
                                            '        ">=2000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_depth" : 1,\n'
                                            '      "latency_target" : 0,\n'
                                            '      "latency_percentile" : '
                                            '100.000000,\n'
                                            '      "latency_window" : 0\n'
                                            '    }\n'
                                            '  ],\n'
                                            '  "disk_util" : [\n'
                                            '    {\n'
                                            '      "name" : "nvme0n1",\n'
                                            '      "read_ios" : 1067722,\n'
                                            '      "write_ios" : 0,\n'
                                            '      "read_merges" : 0,\n'
                                            '      "write_merges" : 0,\n'
                                            '      "read_ticks" : 5364,\n'
                                            '      "write_ticks" : 0,\n'
                                            '      "in_queue" : 5364,\n'
                                            '      "util" : 99.045298\n'
                                            '    }\n'
                                            '  ]\n'
                                            '}\n'
                                            '\n'
                                            'libaio-reference: (groupid=0, '
                                            'jobs=1): err= 0: pid=6017: Wed '
                                            'Oct 26 15:04:12 2022\n'
                                            '  read: IOPS=109k, BW=53.3MiB/s '
                                            '(55.9MB/s)(373MiB/7001msec)\n'
                                            '    slat (nsec): min=1653, '
                                            'max=6341, avg=1761.58, '
                                            'stdev=74.91\n'
                                            '    clat (nsec): min=2775, '
                                            'max=94186, avg=7098.31, '
                                            'stdev=981.59\n'
                                            '     lat (nsec): min=7384, '
                                            'max=95980, avg=8859.89, '
                                            'stdev=985.74\n'
                                            '    clat percentiles (nsec):\n'
                                            '     |  1.00th=[ 6624],  5.00th=[ '
                                            '6816], 10.00th=[ 6880], 20.00th=[ '
                                            '6880],\n'
                                            '     | 30.00th=[ 6944], 40.00th=[ '
                                            '6944], 50.00th=[ 7008], 60.00th=[ '
                                            '7072],\n'
                                            '     | 70.00th=[ 7072], 80.00th=[ '
                                            '7072], 90.00th=[ 7200], 95.00th=[ '
                                            '7264],\n'
                                            '     | 99.00th=[10176], '
                                            '99.50th=[13504], 99.90th=[15424], '
                                            '99.95th=[15680],\n'
                                            '     | 99.99th=[58112]\n'
                                            '   bw (  KiB/s): min=54489, '
                                            'max=54838, per=100.00%, '
                                            'avg=54647.69, stdev=106.50, '
                                            'samples=13\n'
                                            '   iops        : min=108978, '
                                            'max=109676, avg=109295.46, '
                                            'stdev=213.11, samples=13\n'
                                            '  lat (usec)   : 4=0.01%, '
                                            '10=98.87%, 20=1.10%, 50=0.01%, '
                                            '100=0.01%\n'
                                            '  cpu          : usr=15.63%, '
                                            'sys=27.59%, ctx=764759, majf=0, '
                                            'minf=1\n'
                                            '  IO depths    : 1=100.0%, '
                                            '2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, >=64=0.0%\n'
                                            '     submit    : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     complete  : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     issued rwts: '
                                            'total=764757,0,0,0 short=0,0,0,0 '
                                            'dropped=0,0,0,0\n'
                                            '     latency   : target=0, '
                                            'window=0, percentile=100.00%, '
                                            'depth=1\n'
                                            '\n'
                                            'Run status group 0 (all jobs):\n'
                                            '   READ: bw=53.3MiB/s (55.9MB/s), '
                                            '53.3MiB/s-53.3MiB/s '
                                            '(55.9MB/s-55.9MB/s), io=373MiB '
                                            '(392MB), run=7001-7001msec\n'
                                            '\n'
                                            'Disk stats (read/write):\n'
                                            '  nvme0n1: ios=1067722/0, '
                                            'merge=0/0, ticks=5364/0, '
                                            'in_queue=5364, util=99.05%\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_66.output'),
                                  'state': {'begin': 1666782362.77719,
                                            'cmd': 'cat '
                                                   '/tmp/fio-output_512_1_libaio-reference_0.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.8729548454284668,
                                            'end': 1666782363.6501448,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_66.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_66.state')},
                       'cmd_67': {'output': 'rm: cannot remove '
                                            "'/tmp/fio-output_512_1_libaio-reference_1.txt': "
                                            'No such file or directory\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_67.output'),
                                  'state': {'begin': 1666782364.7075977,
                                            'cmd': 'rm '
                                                   '/tmp/fio-output_512_1_libaio-reference_1.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.7930121421813965,
                                            'end': 1666782365.5006099,
                                            'err': 1,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_67.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_67.state')},
                       'cmd_68': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_68.output'),
                                  'state': {'begin': 1666782365.5025218,
                                            'cmd': '/opt/fio/bin/fio '
                                                   '--name="libaio-reference" '
                                                   '--rw="randread" '
                                                   '--size="12G" --iodepth="1" '
                                                   '--bs="512" --direct="1" '
                                                   '--thread="1" '
                                                   '--time_based="1" '
                                                   '--runtime="7" '
                                                   '--ramp_time="3" '
                                                   '--norandommap="1" '
                                                   '--allow_file_create="0" '
                                                   '--ioengine="libaio" '
                                                   '--filename="/dev/nvme0n1" '
                                                   '--output-format="normal,json" '
                                                   '--output="/tmp/fio-output_512_1_libaio-reference_1.txt" '
                                                   '--cpus_allowed="1"',
                                            'cwd': 'None',
                                            'elapsed': 11.134260654449463,
                                            'end': 1666782376.6367824,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_68.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_68.state')},
                       'cmd_69': {'output': 'libaio-reference: (g=0): '
                                            'rw=randread, bs=(R) 512B-512B, '
                                            '(W) 512B-512B, (T) 512B-512B, '
                                            'ioengine=libaio, iodepth=1\n'
                                            'fio-3.32-dirty\n'
                                            'Starting 1 thread\n'
                                            '{\n'
                                            '  "fio version" : '
                                            '"fio-3.32-dirty",\n'
                                            '  "timestamp" : 1666789466,\n'
                                            '  "timestamp_ms" : '
                                            '1666789466728,\n'
                                            '  "time" : "Wed Oct 26 15:04:26 '
                                            '2022",\n'
                                            '  "jobs" : [\n'
                                            '    {\n'
                                            '      "jobname" : '
                                            '"libaio-reference",\n'
                                            '      "groupid" : 0,\n'
                                            '      "error" : 0,\n'
                                            '      "eta" : 0,\n'
                                            '      "elapsed" : 11,\n'
                                            '      "job options" : {\n'
                                            '        "name" : '
                                            '"libaio-reference",\n'
                                            '        "rw" : "randread",\n'
                                            '        "size" : "12G",\n'
                                            '        "iodepth" : "1",\n'
                                            '        "bs" : "512",\n'
                                            '        "direct" : "1",\n'
                                            '        "thread" : "1",\n'
                                            '        "time_based" : "1",\n'
                                            '        "runtime" : "7",\n'
                                            '        "ramp_time" : "3",\n'
                                            '        "norandommap" : "1",\n'
                                            '        "allow_file_create" : '
                                            '"0",\n'
                                            '        "ioengine" : "libaio",\n'
                                            '        "filename" : '
                                            '"/dev/nvme0n1",\n'
                                            '        "cpus_allowed" : "1"\n'
                                            '      },\n'
                                            '      "read" : {\n'
                                            '        "io_bytes" : 393853440,\n'
                                            '        "io_kbytes" : 384622,\n'
                                            '        "bw_bytes" : 56256740,\n'
                                            '        "bw" : 54938,\n'
                                            '        "iops" : 109876.446222,\n'
                                            '        "runtime" : 7001,\n'
                                            '        "total_ios" : 769245,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 1633,\n'
                                            '          "max" : 15660,\n'
                                            '          "mean" : 1733.714902,\n'
                                            '          "stddev" : 59.717360,\n'
                                            '          "N" : 769245\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 581,\n'
                                            '          "max" : 88846,\n'
                                            '          "mean" : 7084.517244,\n'
                                            '          "stddev" : 968.322335,\n'
                                            '          "N" : 769245,\n'
                                            '          "percentile" : {\n'
                                            '            "1.000000" : 6624,\n'
                                            '            "5.000000" : 6816,\n'
                                            '            "10.000000" : 6816,\n'
                                            '            "20.000000" : 6880,\n'
                                            '            "30.000000" : 6880,\n'
                                            '            "40.000000" : 6944,\n'
                                            '            "50.000000" : 7008,\n'
                                            '            "60.000000" : 7008,\n'
                                            '            "70.000000" : 7072,\n'
                                            '            "80.000000" : 7072,\n'
                                            '            "90.000000" : 7200,\n'
                                            '            "95.000000" : 7264,\n'
                                            '            "99.000000" : 10176,\n'
                                            '            "99.500000" : 13504,\n'
                                            '            "99.900000" : 15424,\n'
                                            '            "99.950000" : 15680,\n'
                                            '            "99.990000" : 58112\n'
                                            '          }\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 7294,\n'
                                            '          "max" : 90539,\n'
                                            '          "mean" : 8818.232146,\n'
                                            '          "stddev" : 971.397702,\n'
                                            '          "N" : 769245\n'
                                            '        },\n'
                                            '        "bw_min" : 54727,\n'
                                            '        "bw_max" : 55130,\n'
                                            '        "bw_agg" : 100.000000,\n'
                                            '        "bw_mean" : '
                                            '54952.357143,\n'
                                            '        "bw_dev" : 130.746147,\n'
                                            '        "bw_samples" : 14,\n'
                                            '        "iops_min" : 109454,\n'
                                            '        "iops_max" : 110260,\n'
                                            '        "iops_mean" : '
                                            '109904.714286,\n'
                                            '        "iops_stddev" : '
                                            '261.492294,\n'
                                            '        "iops_samples" : 14\n'
                                            '      },\n'
                                            '      "write" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "trim" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "sync" : {\n'
                                            '        "total_ios" : 0,\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        }\n'
                                            '      },\n'
                                            '      "job_runtime" : 7000,\n'
                                            '      "usr_cpu" : 17.442857,\n'
                                            '      "sys_cpu" : 25.171429,\n'
                                            '      "ctx" : 769247,\n'
                                            '      "majf" : 0,\n'
                                            '      "minf" : 0,\n'
                                            '      "iodepth_level" : {\n'
                                            '        "1" : 100.000000,\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_submit" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_complete" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ns" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.010000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_us" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.010000,\n'
                                            '        "10" : 98.809872,\n'
                                            '        "20" : 1.172708,\n'
                                            '        "50" : 0.010000,\n'
                                            '        "100" : 0.013650,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ms" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000,\n'
                                            '        "2000" : 0.000000,\n'
                                            '        ">=2000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_depth" : 1,\n'
                                            '      "latency_target" : 0,\n'
                                            '      "latency_percentile" : '
                                            '100.000000,\n'
                                            '      "latency_window" : 0\n'
                                            '    }\n'
                                            '  ],\n'
                                            '  "disk_util" : [\n'
                                            '    {\n'
                                            '      "name" : "nvme0n1",\n'
                                            '      "read_ios" : 1099355,\n'
                                            '      "write_ios" : 0,\n'
                                            '      "read_merges" : 0,\n'
                                            '      "write_merges" : 0,\n'
                                            '      "read_ticks" : 5515,\n'
                                            '      "write_ticks" : 0,\n'
                                            '      "in_queue" : 5515,\n'
                                            '      "util" : 99.069123\n'
                                            '    }\n'
                                            '  ]\n'
                                            '}\n'
                                            '\n'
                                            'libaio-reference: (groupid=0, '
                                            'jobs=1): err= 0: pid=6063: Wed '
                                            'Oct 26 15:04:26 2022\n'
                                            '  read: IOPS=110k, BW=53.7MiB/s '
                                            '(56.3MB/s)(376MiB/7001msec)\n'
                                            '    slat (nsec): min=1633, '
                                            'max=15660, avg=1733.71, '
                                            'stdev=59.72\n'
                                            '    clat (nsec): min=581, '
                                            'max=88846, avg=7084.52, '
                                            'stdev=968.32\n'
                                            '     lat (nsec): min=7294, '
                                            'max=90539, avg=8818.23, '
                                            'stdev=971.40\n'
                                            '    clat percentiles (nsec):\n'
                                            '     |  1.00th=[ 6624],  5.00th=[ '
                                            '6816], 10.00th=[ 6816], 20.00th=[ '
                                            '6880],\n'
                                            '     | 30.00th=[ 6880], 40.00th=[ '
                                            '6944], 50.00th=[ 7008], 60.00th=[ '
                                            '7008],\n'
                                            '     | 70.00th=[ 7072], 80.00th=[ '
                                            '7072], 90.00th=[ 7200], 95.00th=[ '
                                            '7264],\n'
                                            '     | 99.00th=[10176], '
                                            '99.50th=[13504], 99.90th=[15424], '
                                            '99.95th=[15680],\n'
                                            '     | 99.99th=[58112]\n'
                                            '   bw (  KiB/s): min=54727, '
                                            'max=55130, per=100.00%, '
                                            'avg=54952.36, stdev=130.75, '
                                            'samples=14\n'
                                            '   iops        : min=109454, '
                                            'max=110260, avg=109904.71, '
                                            'stdev=261.49, samples=14\n'
                                            '  lat (nsec)   : 750=0.01%\n'
                                            '  lat (usec)   : 4=0.01%, '
                                            '10=98.81%, 20=1.17%, 50=0.01%, '
                                            '100=0.01%\n'
                                            '  cpu          : usr=17.44%, '
                                            'sys=25.17%, ctx=769247, majf=0, '
                                            'minf=0\n'
                                            '  IO depths    : 1=100.0%, '
                                            '2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, >=64=0.0%\n'
                                            '     submit    : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     complete  : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     issued rwts: '
                                            'total=769245,0,0,0 short=0,0,0,0 '
                                            'dropped=0,0,0,0\n'
                                            '     latency   : target=0, '
                                            'window=0, percentile=100.00%, '
                                            'depth=1\n'
                                            '\n'
                                            'Run status group 0 (all jobs):\n'
                                            '   READ: bw=53.7MiB/s (56.3MB/s), '
                                            '53.7MiB/s-53.7MiB/s '
                                            '(56.3MB/s-56.3MB/s), io=376MiB '
                                            '(394MB), run=7001-7001msec\n'
                                            '\n'
                                            'Disk stats (read/write):\n'
                                            '  nvme0n1: ios=1099355/0, '
                                            'merge=0/0, ticks=5515/0, '
                                            'in_queue=5515, util=99.07%\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_69.output'),
                                  'state': {'begin': 1666782376.6392207,
                                            'cmd': 'cat '
                                                   '/tmp/fio-output_512_1_libaio-reference_1.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.8122994899749756,
                                            'end': 1666782377.4515202,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_69.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_69.state')},
                       'cmd_70': {'output': 'rm: cannot remove '
                                            "'/tmp/fio-output_512_1_libaio-reference_2.txt': "
                                            'No such file or directory\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_70.output'),
                                  'state': {'begin': 1666782378.5656462,
                                            'cmd': 'rm '
                                                   '/tmp/fio-output_512_1_libaio-reference_2.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.8047864437103271,
                                            'end': 1666782379.3704326,
                                            'err': 1,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_70.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_70.state')},
                       'cmd_71': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_71.output'),
                                  'state': {'begin': 1666782379.3727932,
                                            'cmd': '/opt/fio/bin/fio '
                                                   '--name="libaio-reference" '
                                                   '--rw="randread" '
                                                   '--size="12G" --iodepth="1" '
                                                   '--bs="512" --direct="1" '
                                                   '--thread="1" '
                                                   '--time_based="1" '
                                                   '--runtime="7" '
                                                   '--ramp_time="3" '
                                                   '--norandommap="1" '
                                                   '--allow_file_create="0" '
                                                   '--ioengine="libaio" '
                                                   '--filename="/dev/nvme0n1" '
                                                   '--output-format="normal,json" '
                                                   '--output="/tmp/fio-output_512_1_libaio-reference_2.txt" '
                                                   '--cpus_allowed="1"',
                                            'cwd': 'None',
                                            'elapsed': 11.141789436340332,
                                            'end': 1666782390.5145826,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_71.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_71.state')},
                       'cmd_72': {'output': 'libaio-reference: (g=0): '
                                            'rw=randread, bs=(R) 512B-512B, '
                                            '(W) 512B-512B, (T) 512B-512B, '
                                            'ioengine=libaio, iodepth=1\n'
                                            'fio-3.32-dirty\n'
                                            'Starting 1 thread\n'
                                            '{\n'
                                            '  "fio version" : '
                                            '"fio-3.32-dirty",\n'
                                            '  "timestamp" : 1666789480,\n'
                                            '  "timestamp_ms" : '
                                            '1666789480604,\n'
                                            '  "time" : "Wed Oct 26 15:04:40 '
                                            '2022",\n'
                                            '  "jobs" : [\n'
                                            '    {\n'
                                            '      "jobname" : '
                                            '"libaio-reference",\n'
                                            '      "groupid" : 0,\n'
                                            '      "error" : 0,\n'
                                            '      "eta" : 0,\n'
                                            '      "elapsed" : 11,\n'
                                            '      "job options" : {\n'
                                            '        "name" : '
                                            '"libaio-reference",\n'
                                            '        "rw" : "randread",\n'
                                            '        "size" : "12G",\n'
                                            '        "iodepth" : "1",\n'
                                            '        "bs" : "512",\n'
                                            '        "direct" : "1",\n'
                                            '        "thread" : "1",\n'
                                            '        "time_based" : "1",\n'
                                            '        "runtime" : "7",\n'
                                            '        "ramp_time" : "3",\n'
                                            '        "norandommap" : "1",\n'
                                            '        "allow_file_create" : '
                                            '"0",\n'
                                            '        "ioengine" : "libaio",\n'
                                            '        "filename" : '
                                            '"/dev/nvme0n1",\n'
                                            '        "cpus_allowed" : "1"\n'
                                            '      },\n'
                                            '      "read" : {\n'
                                            '        "io_bytes" : 392696320,\n'
                                            '        "io_kbytes" : 383492,\n'
                                            '        "bw_bytes" : 56099474,\n'
                                            '        "bw" : 54784,\n'
                                            '        "iops" : 109569.285714,\n'
                                            '        "runtime" : 7000,\n'
                                            '        "total_ios" : 766985,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 1613,\n'
                                            '          "max" : 9728,\n'
                                            '          "mean" : 1717.578988,\n'
                                            '          "stddev" : 59.288933,\n'
                                            '          "N" : 766985\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 471,\n'
                                            '          "max" : 114094,\n'
                                            '          "mean" : 7108.687395,\n'
                                            '          "stddev" : '
                                            '1019.143377,\n'
                                            '          "N" : 766985,\n'
                                            '          "percentile" : {\n'
                                            '            "1.000000" : 6624,\n'
                                            '            "5.000000" : 6816,\n'
                                            '            "10.000000" : 6880,\n'
                                            '            "20.000000" : 6880,\n'
                                            '            "30.000000" : 6944,\n'
                                            '            "40.000000" : 6944,\n'
                                            '            "50.000000" : 7008,\n'
                                            '            "60.000000" : 7072,\n'
                                            '            "70.000000" : 7072,\n'
                                            '            "80.000000" : 7136,\n'
                                            '            "90.000000" : 7200,\n'
                                            '            "95.000000" : 7328,\n'
                                            '            "99.000000" : 10176,\n'
                                            '            "99.500000" : 13504,\n'
                                            '            "99.900000" : 15552,\n'
                                            '            "99.950000" : 15680,\n'
                                            '            "99.990000" : 60160\n'
                                            '          }\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 6893,\n'
                                            '          "max" : 115777,\n'
                                            '          "mean" : 8826.266383,\n'
                                            '          "stddev" : '
                                            '1021.839338,\n'
                                            '          "N" : 766985\n'
                                            '        },\n'
                                            '        "bw_min" : 54634,\n'
                                            '        "bw_max" : 55015,\n'
                                            '        "bw_agg" : 100.000000,\n'
                                            '        "bw_mean" : '
                                            '54791.000000,\n'
                                            '        "bw_dev" : 131.780704,\n'
                                            '        "bw_samples" : 14,\n'
                                            '        "iops_min" : 109268,\n'
                                            '        "iops_max" : 110030,\n'
                                            '        "iops_mean" : '
                                            '109582.071429,\n'
                                            '        "iops_stddev" : '
                                            '263.670092,\n'
                                            '        "iops_samples" : 14\n'
                                            '      },\n'
                                            '      "write" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "trim" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "sync" : {\n'
                                            '        "total_ios" : 0,\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        }\n'
                                            '      },\n'
                                            '      "job_runtime" : 7000,\n'
                                            '      "usr_cpu" : 19.400000,\n'
                                            '      "sys_cpu" : 23.571429,\n'
                                            '      "ctx" : 766986,\n'
                                            '      "majf" : 0,\n'
                                            '      "minf" : 0,\n'
                                            '      "iodepth_level" : {\n'
                                            '        "1" : 100.000000,\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_submit" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_complete" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ns" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.010000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_us" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.010000,\n'
                                            '        "10" : 98.838439,\n'
                                            '        "20" : 1.141222,\n'
                                            '        "50" : 0.010000,\n'
                                            '        "100" : 0.016037,\n'
                                            '        "250" : 0.010000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ms" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000,\n'
                                            '        "2000" : 0.000000,\n'
                                            '        ">=2000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_depth" : 1,\n'
                                            '      "latency_target" : 0,\n'
                                            '      "latency_percentile" : '
                                            '100.000000,\n'
                                            '      "latency_window" : 0\n'
                                            '    }\n'
                                            '  ],\n'
                                            '  "disk_util" : [\n'
                                            '    {\n'
                                            '      "name" : "nvme0n1",\n'
                                            '      "read_ios" : 1099551,\n'
                                            '      "write_ios" : 0,\n'
                                            '      "read_merges" : 0,\n'
                                            '      "write_merges" : 0,\n'
                                            '      "read_ticks" : 5504,\n'
                                            '      "write_ticks" : 0,\n'
                                            '      "in_queue" : 5504,\n'
                                            '      "util" : 99.069123\n'
                                            '    }\n'
                                            '  ]\n'
                                            '}\n'
                                            '\n'
                                            'libaio-reference: (groupid=0, '
                                            'jobs=1): err= 0: pid=6110: Wed '
                                            'Oct 26 15:04:40 2022\n'
                                            '  read: IOPS=110k, BW=53.5MiB/s '
                                            '(56.1MB/s)(375MiB/7000msec)\n'
                                            '    slat (nsec): min=1613, '
                                            'max=9728, avg=1717.58, '
                                            'stdev=59.29\n'
                                            '    clat (nsec): min=471, '
                                            'max=114094, avg=7108.69, '
                                            'stdev=1019.14\n'
                                            '     lat (usec): min=6, max=115, '
                                            'avg= 8.83, stdev= 1.02\n'
                                            '    clat percentiles (nsec):\n'
                                            '     |  1.00th=[ 6624],  5.00th=[ '
                                            '6816], 10.00th=[ 6880], 20.00th=[ '
                                            '6880],\n'
                                            '     | 30.00th=[ 6944], 40.00th=[ '
                                            '6944], 50.00th=[ 7008], 60.00th=[ '
                                            '7072],\n'
                                            '     | 70.00th=[ 7072], 80.00th=[ '
                                            '7136], 90.00th=[ 7200], 95.00th=[ '
                                            '7328],\n'
                                            '     | 99.00th=[10176], '
                                            '99.50th=[13504], 99.90th=[15552], '
                                            '99.95th=[15680],\n'
                                            '     | 99.99th=[60160]\n'
                                            '   bw (  KiB/s): min=54634, '
                                            'max=55015, per=100.00%, '
                                            'avg=54791.00, stdev=131.78, '
                                            'samples=14\n'
                                            '   iops        : min=109268, '
                                            'max=110030, avg=109582.07, '
                                            'stdev=263.67, samples=14\n'
                                            '  lat (nsec)   : 500=0.01%\n'
                                            '  lat (usec)   : 4=0.01%, '
                                            '10=98.84%, 20=1.14%, 50=0.01%, '
                                            '100=0.02%\n'
                                            '  lat (usec)   : 250=0.01%\n'
                                            '  cpu          : usr=19.40%, '
                                            'sys=23.57%, ctx=766986, majf=0, '
                                            'minf=0\n'
                                            '  IO depths    : 1=100.0%, '
                                            '2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, >=64=0.0%\n'
                                            '     submit    : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     complete  : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     issued rwts: '
                                            'total=766985,0,0,0 short=0,0,0,0 '
                                            'dropped=0,0,0,0\n'
                                            '     latency   : target=0, '
                                            'window=0, percentile=100.00%, '
                                            'depth=1\n'
                                            '\n'
                                            'Run status group 0 (all jobs):\n'
                                            '   READ: bw=53.5MiB/s (56.1MB/s), '
                                            '53.5MiB/s-53.5MiB/s '
                                            '(56.1MB/s-56.1MB/s), io=375MiB '
                                            '(393MB), run=7000-7000msec\n'
                                            '\n'
                                            'Disk stats (read/write):\n'
                                            '  nvme0n1: ios=1099551/0, '
                                            'merge=0/0, ticks=5504/0, '
                                            'in_queue=5504, util=99.07%\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_72.output'),
                                  'state': {'begin': 1666782390.517187,
                                            'cmd': 'cat '
                                                   '/tmp/fio-output_512_1_libaio-reference_2.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.8540084362030029,
                                            'end': 1666782391.3711956,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_72.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_72.state')},
                       'cmd_73': {'output': 'rm: cannot remove '
                                            "'/tmp/fio-output_512_1_libaio-bdev_aio_0.txt': "
                                            'No such file or directory\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_73.output'),
                                  'state': {'begin': 1666782393.6421034,
                                            'cmd': 'rm '
                                                   '/tmp/fio-output_512_1_libaio-bdev_aio_0.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.8411805629730225,
                                            'end': 1666782394.483284,
                                            'err': 1,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_73.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_73.state')},
                       'cmd_74': {'output': 'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_74.output'),
                                  'state': {'begin': 1666782394.4855397,
                                            'cmd': '/opt/fio/bin/fio '
                                                   '--name="libaio-bdev_aio" '
                                                   '--rw="randread" '
                                                   '--size="12G" --iodepth="1" '
                                                   '--bs="512" --direct="1" '
                                                   '--thread="1" '
                                                   '--time_based="1" '
                                                   '--runtime="7" '
                                                   '--ramp_time="3" '
                                                   '--norandommap="1" '
                                                   '--allow_file_create="0" '
                                                   '--ioengine="spdk_bdev" '
                                                   '--spdk_conf="/tmp/spdk_bdev_aio_foo.conf" '
                                                   '--filename="Nvme0n1" '
                                                   '--output-format="normal,json" '
                                                   '--output="/tmp/fio-output_512_1_libaio-bdev_aio_0.txt" '
                                                   '--cpus_allowed="1"',
                                            'cwd': 'None',
                                            'elapsed': 11.499650239944458,
                                            'end': 1666782405.98519,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_74.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_74.state')},
                       'cmd_75': {'output': 'libaio-bdev_aio: (g=0): '
                                            'rw=randread, bs=(R) 512B-512B, '
                                            '(W) 512B-512B, (T) 512B-512B, '
                                            'ioengine=spdk_bdev, iodepth=1\n'
                                            'fio-3.32-dirty\n'
                                            'Starting 1 thread\n'
                                            '{\n'
                                            '  "fio version" : '
                                            '"fio-3.32-dirty",\n'
                                            '  "timestamp" : 1666789495,\n'
                                            '  "timestamp_ms" : '
                                            '1666789495981,\n'
                                            '  "time" : "Wed Oct 26 15:04:55 '
                                            '2022",\n'
                                            '  "jobs" : [\n'
                                            '    {\n'
                                            '      "jobname" : '
                                            '"libaio-bdev_aio",\n'
                                            '      "groupid" : 0,\n'
                                            '      "error" : 0,\n'
                                            '      "eta" : 0,\n'
                                            '      "elapsed" : 11,\n'
                                            '      "job options" : {\n'
                                            '        "name" : '
                                            '"libaio-bdev_aio",\n'
                                            '        "rw" : "randread",\n'
                                            '        "size" : "12G",\n'
                                            '        "iodepth" : "1",\n'
                                            '        "bs" : "512",\n'
                                            '        "direct" : "1",\n'
                                            '        "thread" : "1",\n'
                                            '        "time_based" : "1",\n'
                                            '        "runtime" : "7",\n'
                                            '        "ramp_time" : "3",\n'
                                            '        "norandommap" : "1",\n'
                                            '        "allow_file_create" : '
                                            '"0",\n'
                                            '        "ioengine" : '
                                            '"spdk_bdev",\n'
                                            '        "spdk_conf" : '
                                            '"/tmp/spdk_bdev_aio_foo.conf",\n'
                                            '        "filename" : "Nvme0n1",\n'
                                            '        "cpus_allowed" : "1"\n'
                                            '      },\n'
                                            '      "read" : {\n'
                                            '        "io_bytes" : 542872576,\n'
                                            '        "io_kbytes" : 530149,\n'
                                            '        "bw_bytes" : 77553225,\n'
                                            '        "bw" : 75735,\n'
                                            '        "iops" : 151471.000000,\n'
                                            '        "runtime" : 7000,\n'
                                            '        "total_ios" : 1060297,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 1733,\n'
                                            '          "max" : 17563,\n'
                                            '          "mean" : 1907.857454,\n'
                                            '          "stddev" : 129.896684,\n'
                                            '          "N" : 1060298\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 230,\n'
                                            '          "max" : 96982,\n'
                                            '          "mean" : 4470.611756,\n'
                                            '          "stddev" : '
                                            '1041.750438,\n'
                                            '          "N" : 1060297,\n'
                                            '          "percentile" : {\n'
                                            '            "1.000000" : 4048,\n'
                                            '            "5.000000" : 4256,\n'
                                            '            "10.000000" : 4256,\n'
                                            '            "20.000000" : 4256,\n'
                                            '            "30.000000" : 4320,\n'
                                            '            "40.000000" : 4320,\n'
                                            '            "50.000000" : 4320,\n'
                                            '            "60.000000" : 4384,\n'
                                            '            "70.000000" : 4384,\n'
                                            '            "80.000000" : 4448,\n'
                                            '            "90.000000" : 4448,\n'
                                            '            "95.000000" : 4640,\n'
                                            '            "99.000000" : 9664,\n'
                                            '            "99.500000" : 10688,\n'
                                            '            "99.900000" : 15296,\n'
                                            '            "99.950000" : 15424,\n'
                                            '            "99.990000" : 49408\n'
                                            '          }\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 5640,\n'
                                            '          "max" : 98935,\n'
                                            '          "mean" : 6378.469139,\n'
                                            '          "stddev" : '
                                            '1051.660338,\n'
                                            '          "N" : 1060297\n'
                                            '        },\n'
                                            '        "bw_min" : 75451,\n'
                                            '        "bw_max" : 76128,\n'
                                            '        "bw_agg" : 100.000000,\n'
                                            '        "bw_mean" : '
                                            '75738.142857,\n'
                                            '        "bw_dev" : 208.255707,\n'
                                            '        "bw_samples" : 14,\n'
                                            '        "iops_min" : 150902,\n'
                                            '        "iops_max" : 152256,\n'
                                            '        "iops_mean" : '
                                            '151476.285714,\n'
                                            '        "iops_stddev" : '
                                            '416.511414,\n'
                                            '        "iops_samples" : 14\n'
                                            '      },\n'
                                            '      "write" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "trim" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "sync" : {\n'
                                            '        "total_ios" : 0,\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        }\n'
                                            '      },\n'
                                            '      "job_runtime" : 7000,\n'
                                            '      "usr_cpu" : 77.700000,\n'
                                            '      "sys_cpu" : 22.271429,\n'
                                            '      "ctx" : 10,\n'
                                            '      "majf" : 0,\n'
                                            '      "minf" : 0,\n'
                                            '      "iodepth_level" : {\n'
                                            '        "1" : 100.000000,\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_submit" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_complete" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ns" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.010000,\n'
                                            '        "500" : 0.010000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_us" : {\n'
                                            '        "2" : 0.010000,\n'
                                            '        "4" : 0.755637,\n'
                                            '        "10" : 98.431383,\n'
                                            '        "20" : 0.798361,\n'
                                            '        "50" : 0.010000,\n'
                                            '        "100" : 0.010000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ms" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000,\n'
                                            '        "2000" : 0.000000,\n'
                                            '        ">=2000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_depth" : 1,\n'
                                            '      "latency_target" : 0,\n'
                                            '      "latency_percentile" : '
                                            '100.000000,\n'
                                            '      "latency_window" : 0\n'
                                            '    }\n'
                                            '  ]\n'
                                            '}\n'
                                            '\n'
                                            'libaio-bdev_aio: (groupid=0, '
                                            'jobs=1): err= 0: pid=6168: Wed '
                                            'Oct 26 15:04:55 2022\n'
                                            '  read: IOPS=151k, BW=74.0MiB/s '
                                            '(77.6MB/s)(518MiB/7000msec)\n'
                                            '    slat (nsec): min=1733, '
                                            'max=17563, avg=1907.86, '
                                            'stdev=129.90\n'
                                            '    clat (nsec): min=230, '
                                            'max=96982, avg=4470.61, '
                                            'stdev=1041.75\n'
                                            '     lat (nsec): min=5640, '
                                            'max=98935, avg=6378.47, '
                                            'stdev=1051.66\n'
                                            '    clat percentiles (nsec):\n'
                                            '     |  1.00th=[ 4048],  5.00th=[ '
                                            '4256], 10.00th=[ 4256], 20.00th=[ '
                                            '4256],\n'
                                            '     | 30.00th=[ 4320], 40.00th=[ '
                                            '4320], 50.00th=[ 4320], 60.00th=[ '
                                            '4384],\n'
                                            '     | 70.00th=[ 4384], 80.00th=[ '
                                            '4448], 90.00th=[ 4448], 95.00th=[ '
                                            '4640],\n'
                                            '     | 99.00th=[ 9664], '
                                            '99.50th=[10688], 99.90th=[15296], '
                                            '99.95th=[15424],\n'
                                            '     | 99.99th=[49408]\n'
                                            '   bw (  KiB/s): min=75451, '
                                            'max=76128, per=100.00%, '
                                            'avg=75738.14, stdev=208.26, '
                                            'samples=14\n'
                                            '   iops        : min=150902, '
                                            'max=152256, avg=151476.29, '
                                            'stdev=416.51, samples=14\n'
                                            '  lat (nsec)   : 250=0.01%, '
                                            '500=0.01%\n'
                                            '  lat (usec)   : 2=0.01%, '
                                            '4=0.76%, 10=98.43%, 20=0.80%, '
                                            '50=0.01%\n'
                                            '  lat (usec)   : 100=0.01%\n'
                                            '  cpu          : usr=77.70%, '
                                            'sys=22.27%, ctx=10, majf=0, '
                                            'minf=0\n'
                                            '  IO depths    : 1=100.0%, '
                                            '2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, >=64=0.0%\n'
                                            '     submit    : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     complete  : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     issued rwts: '
                                            'total=1060297,0,0,0 short=0,0,0,0 '
                                            'dropped=0,0,0,0\n'
                                            '     latency   : target=0, '
                                            'window=0, percentile=100.00%, '
                                            'depth=1\n'
                                            '\n'
                                            'Run status group 0 (all jobs):\n'
                                            '   READ: bw=74.0MiB/s (77.6MB/s), '
                                            '74.0MiB/s-74.0MiB/s '
                                            '(77.6MB/s-77.6MB/s), io=518MiB '
                                            '(543MB), run=7000-7000msec\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_75.output'),
                                  'state': {'begin': 1666782405.9877403,
                                            'cmd': 'cat '
                                                   '/tmp/fio-output_512_1_libaio-bdev_aio_0.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.8320722579956055,
                                            'end': 1666782406.8198125,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_75.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_75.state')},
                       'cmd_76': {'output': 'rm: cannot remove '
                                            "'/tmp/fio-output_512_1_libaio-bdev_aio_1.txt': "
                                            'No such file or directory\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_76.output'),
                                  'state': {'begin': 1666782408.9964058,
                                            'cmd': 'rm '
                                                   '/tmp/fio-output_512_1_libaio-bdev_aio_1.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.7969005107879639,
                                            'end': 1666782409.7933064,
                                            'err': 1,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_76.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_76.state')},
                       'cmd_77': {'output': 'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_77.output'),
                                  'state': {'begin': 1666782409.7953362,
                                            'cmd': '/opt/fio/bin/fio '
                                                   '--name="libaio-bdev_aio" '
                                                   '--rw="randread" '
                                                   '--size="12G" --iodepth="1" '
                                                   '--bs="512" --direct="1" '
                                                   '--thread="1" '
                                                   '--time_based="1" '
                                                   '--runtime="7" '
                                                   '--ramp_time="3" '
                                                   '--norandommap="1" '
                                                   '--allow_file_create="0" '
                                                   '--ioengine="spdk_bdev" '
                                                   '--spdk_conf="/tmp/spdk_bdev_aio_foo.conf" '
                                                   '--filename="Nvme0n1" '
                                                   '--output-format="normal,json" '
                                                   '--output="/tmp/fio-output_512_1_libaio-bdev_aio_1.txt" '
                                                   '--cpus_allowed="1"',
                                            'cwd': 'None',
                                            'elapsed': 11.568880081176758,
                                            'end': 1666782421.3642163,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_77.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_77.state')},
                       'cmd_78': {'output': 'libaio-bdev_aio: (g=0): '
                                            'rw=randread, bs=(R) 512B-512B, '
                                            '(W) 512B-512B, (T) 512B-512B, '
                                            'ioengine=spdk_bdev, iodepth=1\n'
                                            'fio-3.32-dirty\n'
                                            'Starting 1 thread\n'
                                            '{\n'
                                            '  "fio version" : '
                                            '"fio-3.32-dirty",\n'
                                            '  "timestamp" : 1666789511,\n'
                                            '  "timestamp_ms" : '
                                            '1666789511342,\n'
                                            '  "time" : "Wed Oct 26 15:05:11 '
                                            '2022",\n'
                                            '  "jobs" : [\n'
                                            '    {\n'
                                            '      "jobname" : '
                                            '"libaio-bdev_aio",\n'
                                            '      "groupid" : 0,\n'
                                            '      "error" : 0,\n'
                                            '      "eta" : 0,\n'
                                            '      "elapsed" : 11,\n'
                                            '      "job options" : {\n'
                                            '        "name" : '
                                            '"libaio-bdev_aio",\n'
                                            '        "rw" : "randread",\n'
                                            '        "size" : "12G",\n'
                                            '        "iodepth" : "1",\n'
                                            '        "bs" : "512",\n'
                                            '        "direct" : "1",\n'
                                            '        "thread" : "1",\n'
                                            '        "time_based" : "1",\n'
                                            '        "runtime" : "7",\n'
                                            '        "ramp_time" : "3",\n'
                                            '        "norandommap" : "1",\n'
                                            '        "allow_file_create" : '
                                            '"0",\n'
                                            '        "ioengine" : '
                                            '"spdk_bdev",\n'
                                            '        "spdk_conf" : '
                                            '"/tmp/spdk_bdev_aio_foo.conf",\n'
                                            '        "filename" : "Nvme0n1",\n'
                                            '        "cpus_allowed" : "1"\n'
                                            '      },\n'
                                            '      "read" : {\n'
                                            '        "io_bytes" : 544499200,\n'
                                            '        "io_kbytes" : 531737,\n'
                                            '        "bw_bytes" : 77774489,\n'
                                            '        "bw" : 75951,\n'
                                            '        "iops" : 151903.299529,\n'
                                            '        "runtime" : 7001,\n'
                                            '        "total_ios" : 1063475,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 1753,\n'
                                            '          "max" : 16831,\n'
                                            '          "mean" : 1900.398060,\n'
                                            '          "stddev" : 139.445188,\n'
                                            '          "N" : 1063475\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 210,\n'
                                            '          "max" : 77395,\n'
                                            '          "mean" : 4459.766281,\n'
                                            '          "stddev" : '
                                            '1077.834294,\n'
                                            '          "N" : 1063475,\n'
                                            '          "percentile" : {\n'
                                            '            "1.000000" : 4048,\n'
                                            '            "5.000000" : 4192,\n'
                                            '            "10.000000" : 4256,\n'
                                            '            "20.000000" : 4256,\n'
                                            '            "30.000000" : 4256,\n'
                                            '            "40.000000" : 4320,\n'
                                            '            "50.000000" : 4320,\n'
                                            '            "60.000000" : 4384,\n'
                                            '            "70.000000" : 4384,\n'
                                            '            "80.000000" : 4384,\n'
                                            '            "90.000000" : 4448,\n'
                                            '            "95.000000" : 4576,\n'
                                            '            "99.000000" : 9664,\n'
                                            '            "99.500000" : 10560,\n'
                                            '            "99.900000" : 15296,\n'
                                            '            "99.950000" : 15424,\n'
                                            '            "99.990000" : 57088\n'
                                            '          }\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 5620,\n'
                                            '          "max" : 79289,\n'
                                            '          "mean" : 6360.164341,\n'
                                            '          "stddev" : '
                                            '1089.019246,\n'
                                            '          "N" : 1063475\n'
                                            '        },\n'
                                            '        "bw_min" : 75781,\n'
                                            '        "bw_max" : 76168,\n'
                                            '        "bw_agg" : 100.000000,\n'
                                            '        "bw_mean" : '
                                            '75991.692308,\n'
                                            '        "bw_dev" : 122.716058,\n'
                                            '        "bw_samples" : 13,\n'
                                            '        "iops_min" : 151562,\n'
                                            '        "iops_max" : 152336,\n'
                                            '        "iops_mean" : '
                                            '151983.461538,\n'
                                            '        "iops_stddev" : '
                                            '245.379507,\n'
                                            '        "iops_samples" : 13\n'
                                            '      },\n'
                                            '      "write" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "trim" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "sync" : {\n'
                                            '        "total_ios" : 0,\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        }\n'
                                            '      },\n'
                                            '      "job_runtime" : 7000,\n'
                                            '      "usr_cpu" : 74.900000,\n'
                                            '      "sys_cpu" : 25.071429,\n'
                                            '      "ctx" : 10,\n'
                                            '      "majf" : 0,\n'
                                            '      "minf" : 0,\n'
                                            '      "iodepth_level" : {\n'
                                            '        "1" : 100.000000,\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_submit" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_complete" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ns" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.010000,\n'
                                            '        "500" : 0.010000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.010000\n'
                                            '      },\n'
                                            '      "latency_us" : {\n'
                                            '        "2" : 0.010000,\n'
                                            '        "4" : 0.799643,\n'
                                            '        "10" : 98.374292,\n'
                                            '        "20" : 0.809046,\n'
                                            '        "50" : 0.010000,\n'
                                            '        "100" : 0.012130,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ms" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000,\n'
                                            '        "2000" : 0.000000,\n'
                                            '        ">=2000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_depth" : 1,\n'
                                            '      "latency_target" : 0,\n'
                                            '      "latency_percentile" : '
                                            '100.000000,\n'
                                            '      "latency_window" : 0\n'
                                            '    }\n'
                                            '  ]\n'
                                            '}\n'
                                            '\n'
                                            'libaio-bdev_aio: (groupid=0, '
                                            'jobs=1): err= 0: pid=6226: Wed '
                                            'Oct 26 15:05:11 2022\n'
                                            '  read: IOPS=152k, BW=74.2MiB/s '
                                            '(77.8MB/s)(519MiB/7001msec)\n'
                                            '    slat (nsec): min=1753, '
                                            'max=16831, avg=1900.40, '
                                            'stdev=139.45\n'
                                            '    clat (nsec): min=210, '
                                            'max=77395, avg=4459.77, '
                                            'stdev=1077.83\n'
                                            '     lat (nsec): min=5620, '
                                            'max=79289, avg=6360.16, '
                                            'stdev=1089.02\n'
                                            '    clat percentiles (nsec):\n'
                                            '     |  1.00th=[ 4048],  5.00th=[ '
                                            '4192], 10.00th=[ 4256], 20.00th=[ '
                                            '4256],\n'
                                            '     | 30.00th=[ 4256], 40.00th=[ '
                                            '4320], 50.00th=[ 4320], 60.00th=[ '
                                            '4384],\n'
                                            '     | 70.00th=[ 4384], 80.00th=[ '
                                            '4384], 90.00th=[ 4448], 95.00th=[ '
                                            '4576],\n'
                                            '     | 99.00th=[ 9664], '
                                            '99.50th=[10560], 99.90th=[15296], '
                                            '99.95th=[15424],\n'
                                            '     | 99.99th=[57088]\n'
                                            '   bw (  KiB/s): min=75781, '
                                            'max=76168, per=100.00%, '
                                            'avg=75991.69, stdev=122.72, '
                                            'samples=13\n'
                                            '   iops        : min=151562, '
                                            'max=152336, avg=151983.46, '
                                            'stdev=245.38, samples=13\n'
                                            '  lat (nsec)   : 250=0.01%, '
                                            '500=0.01%, 1000=0.01%\n'
                                            '  lat (usec)   : 2=0.01%, '
                                            '4=0.80%, 10=98.37%, 20=0.81%, '
                                            '50=0.01%\n'
                                            '  lat (usec)   : 100=0.01%\n'
                                            '  cpu          : usr=74.90%, '
                                            'sys=25.07%, ctx=10, majf=0, '
                                            'minf=0\n'
                                            '  IO depths    : 1=100.0%, '
                                            '2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, >=64=0.0%\n'
                                            '     submit    : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     complete  : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     issued rwts: '
                                            'total=1063475,0,0,0 short=0,0,0,0 '
                                            'dropped=0,0,0,0\n'
                                            '     latency   : target=0, '
                                            'window=0, percentile=100.00%, '
                                            'depth=1\n'
                                            '\n'
                                            'Run status group 0 (all jobs):\n'
                                            '   READ: bw=74.2MiB/s (77.8MB/s), '
                                            '74.2MiB/s-74.2MiB/s '
                                            '(77.8MB/s-77.8MB/s), io=519MiB '
                                            '(544MB), run=7001-7001msec\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_78.output'),
                                  'state': {'begin': 1666782421.3675125,
                                            'cmd': 'cat '
                                                   '/tmp/fio-output_512_1_libaio-bdev_aio_1.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.8112480640411377,
                                            'end': 1666782422.1787605,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_78.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_78.state')},
                       'cmd_79': {'output': 'rm: cannot remove '
                                            "'/tmp/fio-output_512_1_libaio-bdev_aio_2.txt': "
                                            'No such file or directory\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_79.output'),
                                  'state': {'begin': 1666782424.267234,
                                            'cmd': 'rm '
                                                   '/tmp/fio-output_512_1_libaio-bdev_aio_2.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.9013993740081787,
                                            'end': 1666782425.1686335,
                                            'err': 1,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_79.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_79.state')},
                       'cmd_80': {'output': 'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_80.output'),
                                  'state': {'begin': 1666782425.1756573,
                                            'cmd': '/opt/fio/bin/fio '
                                                   '--name="libaio-bdev_aio" '
                                                   '--rw="randread" '
                                                   '--size="12G" --iodepth="1" '
                                                   '--bs="512" --direct="1" '
                                                   '--thread="1" '
                                                   '--time_based="1" '
                                                   '--runtime="7" '
                                                   '--ramp_time="3" '
                                                   '--norandommap="1" '
                                                   '--allow_file_create="0" '
                                                   '--ioengine="spdk_bdev" '
                                                   '--spdk_conf="/tmp/spdk_bdev_aio_foo.conf" '
                                                   '--filename="Nvme0n1" '
                                                   '--output-format="normal,json" '
                                                   '--output="/tmp/fio-output_512_1_libaio-bdev_aio_2.txt" '
                                                   '--cpus_allowed="1"',
                                            'cwd': 'None',
                                            'elapsed': 11.547270059585571,
                                            'end': 1666782436.7229273,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_80.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_80.state')},
                       'cmd_81': {'output': 'libaio-bdev_aio: (g=0): '
                                            'rw=randread, bs=(R) 512B-512B, '
                                            '(W) 512B-512B, (T) 512B-512B, '
                                            'ioengine=spdk_bdev, iodepth=1\n'
                                            'fio-3.32-dirty\n'
                                            'Starting 1 thread\n'
                                            '{\n'
                                            '  "fio version" : '
                                            '"fio-3.32-dirty",\n'
                                            '  "timestamp" : 1666789526,\n'
                                            '  "timestamp_ms" : '
                                            '1666789526714,\n'
                                            '  "time" : "Wed Oct 26 15:05:26 '
                                            '2022",\n'
                                            '  "jobs" : [\n'
                                            '    {\n'
                                            '      "jobname" : '
                                            '"libaio-bdev_aio",\n'
                                            '      "groupid" : 0,\n'
                                            '      "error" : 0,\n'
                                            '      "eta" : 0,\n'
                                            '      "elapsed" : 11,\n'
                                            '      "job options" : {\n'
                                            '        "name" : '
                                            '"libaio-bdev_aio",\n'
                                            '        "rw" : "randread",\n'
                                            '        "size" : "12G",\n'
                                            '        "iodepth" : "1",\n'
                                            '        "bs" : "512",\n'
                                            '        "direct" : "1",\n'
                                            '        "thread" : "1",\n'
                                            '        "time_based" : "1",\n'
                                            '        "runtime" : "7",\n'
                                            '        "ramp_time" : "3",\n'
                                            '        "norandommap" : "1",\n'
                                            '        "allow_file_create" : '
                                            '"0",\n'
                                            '        "ioengine" : '
                                            '"spdk_bdev",\n'
                                            '        "spdk_conf" : '
                                            '"/tmp/spdk_bdev_aio_foo.conf",\n'
                                            '        "filename" : "Nvme0n1",\n'
                                            '        "cpus_allowed" : "1"\n'
                                            '      },\n'
                                            '      "read" : {\n'
                                            '        "io_bytes" : 542993920,\n'
                                            '        "io_kbytes" : 530267,\n'
                                            '        "bw_bytes" : 77559480,\n'
                                            '        "bw" : 75741,\n'
                                            '        "iops" : 151483.359520,\n'
                                            '        "runtime" : 7001,\n'
                                            '        "total_ios" : 1060535,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 1733,\n'
                                            '          "max" : 17993,\n'
                                            '          "mean" : 1902.296994,\n'
                                            '          "stddev" : 142.855709,\n'
                                            '          "N" : 1060535\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 220,\n'
                                            '          "max" : 99156,\n'
                                            '          "mean" : 4465.624404,\n'
                                            '          "stddev" : '
                                            '1078.405463,\n'
                                            '          "N" : 1060535,\n'
                                            '          "percentile" : {\n'
                                            '            "1.000000" : 4048,\n'
                                            '            "5.000000" : 4256,\n'
                                            '            "10.000000" : 4256,\n'
                                            '            "20.000000" : 4256,\n'
                                            '            "30.000000" : 4320,\n'
                                            '            "40.000000" : 4320,\n'
                                            '            "50.000000" : 4320,\n'
                                            '            "60.000000" : 4384,\n'
                                            '            "70.000000" : 4384,\n'
                                            '            "80.000000" : 4384,\n'
                                            '            "90.000000" : 4448,\n'
                                            '            "95.000000" : 4576,\n'
                                            '            "99.000000" : 9664,\n'
                                            '            "99.500000" : 10688,\n'
                                            '            "99.900000" : 15296,\n'
                                            '            "99.950000" : 15424,\n'
                                            '            "99.990000" : 57088\n'
                                            '          }\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 5630,\n'
                                            '          "max" : 101070,\n'
                                            '          "mean" : 6367.921398,\n'
                                            '          "stddev" : '
                                            '1089.091086,\n'
                                            '          "N" : 1060535\n'
                                            '        },\n'
                                            '        "bw_min" : 75607,\n'
                                            '        "bw_max" : 75920,\n'
                                            '        "bw_agg" : 100.000000,\n'
                                            '        "bw_mean" : '
                                            '75783.928571,\n'
                                            '        "bw_dev" : 71.611628,\n'
                                            '        "bw_samples" : 14,\n'
                                            '        "iops_min" : 151214,\n'
                                            '        "iops_max" : 151838,\n'
                                            '        "iops_mean" : '
                                            '151567.857143,\n'
                                            '        "iops_stddev" : '
                                            '142.936703,\n'
                                            '        "iops_samples" : 14\n'
                                            '      },\n'
                                            '      "write" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "trim" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "sync" : {\n'
                                            '        "total_ios" : 0,\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        }\n'
                                            '      },\n'
                                            '      "job_runtime" : 7000,\n'
                                            '      "usr_cpu" : 76.042857,\n'
                                            '      "sys_cpu" : 23.928571,\n'
                                            '      "ctx" : 10,\n'
                                            '      "majf" : 0,\n'
                                            '      "minf" : 0,\n'
                                            '      "iodepth_level" : {\n'
                                            '        "1" : 100.000000,\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_submit" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_complete" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ns" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.010000,\n'
                                            '        "500" : 0.010000,\n'
                                            '        "750" : 0.010000,\n'
                                            '        "1000" : 0.010000\n'
                                            '      },\n'
                                            '      "latency_us" : {\n'
                                            '        "2" : 0.010000,\n'
                                            '        "4" : 0.758202,\n'
                                            '        "10" : 98.419666,\n'
                                            '        "20" : 0.804028,\n'
                                            '        "50" : 0.010000,\n'
                                            '        "100" : 0.011881,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ms" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000,\n'
                                            '        "2000" : 0.000000,\n'
                                            '        ">=2000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_depth" : 1,\n'
                                            '      "latency_target" : 0,\n'
                                            '      "latency_percentile" : '
                                            '100.000000,\n'
                                            '      "latency_window" : 0\n'
                                            '    }\n'
                                            '  ]\n'
                                            '}\n'
                                            '\n'
                                            'libaio-bdev_aio: (groupid=0, '
                                            'jobs=1): err= 0: pid=6285: Wed '
                                            'Oct 26 15:05:26 2022\n'
                                            '  read: IOPS=151k, BW=74.0MiB/s '
                                            '(77.6MB/s)(518MiB/7001msec)\n'
                                            '    slat (nsec): min=1733, '
                                            'max=17993, avg=1902.30, '
                                            'stdev=142.86\n'
                                            '    clat (nsec): min=220, '
                                            'max=99156, avg=4465.62, '
                                            'stdev=1078.41\n'
                                            '     lat (usec): min=5, max=101, '
                                            'avg= 6.37, stdev= 1.09\n'
                                            '    clat percentiles (nsec):\n'
                                            '     |  1.00th=[ 4048],  5.00th=[ '
                                            '4256], 10.00th=[ 4256], 20.00th=[ '
                                            '4256],\n'
                                            '     | 30.00th=[ 4320], 40.00th=[ '
                                            '4320], 50.00th=[ 4320], 60.00th=[ '
                                            '4384],\n'
                                            '     | 70.00th=[ 4384], 80.00th=[ '
                                            '4384], 90.00th=[ 4448], 95.00th=[ '
                                            '4576],\n'
                                            '     | 99.00th=[ 9664], '
                                            '99.50th=[10688], 99.90th=[15296], '
                                            '99.95th=[15424],\n'
                                            '     | 99.99th=[57088]\n'
                                            '   bw (  KiB/s): min=75607, '
                                            'max=75920, per=100.00%, '
                                            'avg=75783.93, stdev=71.61, '
                                            'samples=14\n'
                                            '   iops        : min=151214, '
                                            'max=151838, avg=151567.86, '
                                            'stdev=142.94, samples=14\n'
                                            '  lat (nsec)   : 250=0.01%, '
                                            '500=0.01%, 750=0.01%, 1000=0.01%\n'
                                            '  lat (usec)   : 2=0.01%, '
                                            '4=0.76%, 10=98.42%, 20=0.80%, '
                                            '50=0.01%\n'
                                            '  lat (usec)   : 100=0.01%\n'
                                            '  cpu          : usr=76.04%, '
                                            'sys=23.93%, ctx=10, majf=0, '
                                            'minf=0\n'
                                            '  IO depths    : 1=100.0%, '
                                            '2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, >=64=0.0%\n'
                                            '     submit    : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     complete  : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     issued rwts: '
                                            'total=1060535,0,0,0 short=0,0,0,0 '
                                            'dropped=0,0,0,0\n'
                                            '     latency   : target=0, '
                                            'window=0, percentile=100.00%, '
                                            'depth=1\n'
                                            '\n'
                                            'Run status group 0 (all jobs):\n'
                                            '   READ: bw=74.0MiB/s (77.6MB/s), '
                                            '74.0MiB/s-74.0MiB/s '
                                            '(77.6MB/s-77.6MB/s), io=518MiB '
                                            '(543MB), run=7001-7001msec\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_81.output'),
                                  'state': {'begin': 1666782436.729881,
                                            'cmd': 'cat '
                                                   '/tmp/fio-output_512_1_libaio-bdev_aio_2.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.813136100769043,
                                            'end': 1666782437.5430171,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_81.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_81.state')},
                       'cmd_82': {'output': 'rm: cannot remove '
                                            "'/tmp/fio-output_512_1_libaio-bdev_xnvme_0.txt': "
                                            'No such file or directory\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_82.output'),
                                  'state': {'begin': 1666782439.7651637,
                                            'cmd': 'rm '
                                                   '/tmp/fio-output_512_1_libaio-bdev_xnvme_0.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.8066737651824951,
                                            'end': 1666782440.5718374,
                                            'err': 1,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_82.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_82.state')},
                       'cmd_83': {'output': 'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_83.output'),
                                  'state': {'begin': 1666782440.5739655,
                                            'cmd': '/opt/fio/bin/fio '
                                                   '--name="libaio-bdev_xnvme" '
                                                   '--rw="randread" '
                                                   '--size="12G" --iodepth="1" '
                                                   '--bs="512" --direct="1" '
                                                   '--thread="1" '
                                                   '--time_based="1" '
                                                   '--runtime="7" '
                                                   '--ramp_time="3" '
                                                   '--norandommap="1" '
                                                   '--allow_file_create="0" '
                                                   '--ioengine="spdk_bdev" '
                                                   '--spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" '
                                                   '--filename="Nvme0n1" '
                                                   '--output-format="normal,json" '
                                                   '--output="/tmp/fio-output_512_1_libaio-bdev_xnvme_0.txt" '
                                                   '--cpus_allowed="1"',
                                            'cwd': 'None',
                                            'elapsed': 11.549588680267334,
                                            'end': 1666782452.1235542,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_83.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_83.state')},
                       'cmd_84': {'output': 'libaio-bdev_xnvme: (g=0): '
                                            'rw=randread, bs=(R) 512B-512B, '
                                            '(W) 512B-512B, (T) 512B-512B, '
                                            'ioengine=spdk_bdev, iodepth=1\n'
                                            'fio-3.32-dirty\n'
                                            'Starting 1 thread\n'
                                            '{\n'
                                            '  "fio version" : '
                                            '"fio-3.32-dirty",\n'
                                            '  "timestamp" : 1666789542,\n'
                                            '  "timestamp_ms" : '
                                            '1666789542121,\n'
                                            '  "time" : "Wed Oct 26 15:05:42 '
                                            '2022",\n'
                                            '  "jobs" : [\n'
                                            '    {\n'
                                            '      "jobname" : '
                                            '"libaio-bdev_xnvme",\n'
                                            '      "groupid" : 0,\n'
                                            '      "error" : 0,\n'
                                            '      "eta" : 0,\n'
                                            '      "elapsed" : 11,\n'
                                            '      "job options" : {\n'
                                            '        "name" : '
                                            '"libaio-bdev_xnvme",\n'
                                            '        "rw" : "randread",\n'
                                            '        "size" : "12G",\n'
                                            '        "iodepth" : "1",\n'
                                            '        "bs" : "512",\n'
                                            '        "direct" : "1",\n'
                                            '        "thread" : "1",\n'
                                            '        "time_based" : "1",\n'
                                            '        "runtime" : "7",\n'
                                            '        "ramp_time" : "3",\n'
                                            '        "norandommap" : "1",\n'
                                            '        "allow_file_create" : '
                                            '"0",\n'
                                            '        "ioengine" : '
                                            '"spdk_bdev",\n'
                                            '        "spdk_conf" : '
                                            '"/tmp/spdk_bdev_xnvme_libaio.conf",\n'
                                            '        "filename" : "Nvme0n1",\n'
                                            '        "cpus_allowed" : "1"\n'
                                            '      },\n'
                                            '      "read" : {\n'
                                            '        "io_bytes" : 523051520,\n'
                                            '        "io_kbytes" : 510792,\n'
                                            '        "bw_bytes" : 74710972,\n'
                                            '        "bw" : 72959,\n'
                                            '        "iops" : 145919.868590,\n'
                                            '        "runtime" : 7001,\n'
                                            '        "total_ios" : 1021585,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 1753,\n'
                                            '          "max" : 40205,\n'
                                            '          "mean" : 1902.474167,\n'
                                            '          "stddev" : 112.382836,\n'
                                            '          "N" : 1021585\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 591,\n'
                                            '          "max" : 101370,\n'
                                            '          "mean" : 4716.904850,\n'
                                            '          "stddev" : '
                                            '1098.648880,\n'
                                            '          "N" : 1021585,\n'
                                            '          "percentile" : {\n'
                                            '            "1.000000" : 4384,\n'
                                            '            "5.000000" : 4384,\n'
                                            '            "10.000000" : 4448,\n'
                                            '            "20.000000" : 4448,\n'
                                            '            "30.000000" : 4448,\n'
                                            '            "40.000000" : 4640,\n'
                                            '            "50.000000" : 4640,\n'
                                            '            "60.000000" : 4640,\n'
                                            '            "70.000000" : 4704,\n'
                                            '            "80.000000" : 4704,\n'
                                            '            "90.000000" : 4704,\n'
                                            '            "95.000000" : 4896,\n'
                                            '            "99.000000" : 9664,\n'
                                            '            "99.500000" : 10944,\n'
                                            '            "99.900000" : 15296,\n'
                                            '            "99.950000" : 15424,\n'
                                            '            "99.990000" : 58624\n'
                                            '          }\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 5871,\n'
                                            '          "max" : 103304,\n'
                                            '          "mean" : 6619.379017,\n'
                                            '          "stddev" : '
                                            '1103.785829,\n'
                                            '          "N" : 1021585\n'
                                            '        },\n'
                                            '        "bw_min" : 72756,\n'
                                            '        "bw_max" : 73193,\n'
                                            '        "bw_agg" : 100.000000,\n'
                                            '        "bw_mean" : '
                                            '72972.928571,\n'
                                            '        "bw_dev" : 114.125750,\n'
                                            '        "bw_samples" : 14,\n'
                                            '        "iops_min" : 145512,\n'
                                            '        "iops_max" : 146386,\n'
                                            '        "iops_mean" : '
                                            '145945.857143,\n'
                                            '        "iops_stddev" : '
                                            '228.251500,\n'
                                            '        "iops_samples" : 14\n'
                                            '      },\n'
                                            '      "write" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "trim" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "sync" : {\n'
                                            '        "total_ios" : 0,\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        }\n'
                                            '      },\n'
                                            '      "job_runtime" : 7000,\n'
                                            '      "usr_cpu" : 54.728571,\n'
                                            '      "sys_cpu" : 45.242857,\n'
                                            '      "ctx" : 10,\n'
                                            '      "majf" : 0,\n'
                                            '      "minf" : 0,\n'
                                            '      "iodepth_level" : {\n'
                                            '        "1" : 100.000000,\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_submit" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_complete" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ns" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.010000,\n'
                                            '        "1000" : 0.010000\n'
                                            '      },\n'
                                            '      "latency_us" : {\n'
                                            '        "2" : 0.010000,\n'
                                            '        "4" : 0.017913,\n'
                                            '        "10" : 99.114122,\n'
                                            '        "20" : 0.844962,\n'
                                            '        "50" : 0.010000,\n'
                                            '        "100" : 0.014389,\n'
                                            '        "250" : 0.010000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ms" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000,\n'
                                            '        "2000" : 0.000000,\n'
                                            '        ">=2000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_depth" : 1,\n'
                                            '      "latency_target" : 0,\n'
                                            '      "latency_percentile" : '
                                            '100.000000,\n'
                                            '      "latency_window" : 0\n'
                                            '    }\n'
                                            '  ]\n'
                                            '}\n'
                                            '\n'
                                            'libaio-bdev_xnvme: (groupid=0, '
                                            'jobs=1): err= 0: pid=6343: Wed '
                                            'Oct 26 15:05:42 2022\n'
                                            '  read: IOPS=146k, BW=71.2MiB/s '
                                            '(74.7MB/s)(499MiB/7001msec)\n'
                                            '    slat (nsec): min=1753, '
                                            'max=40205, avg=1902.47, '
                                            'stdev=112.38\n'
                                            '    clat (nsec): min=591, '
                                            'max=101370, avg=4716.90, '
                                            'stdev=1098.65\n'
                                            '     lat (usec): min=5, max=103, '
                                            'avg= 6.62, stdev= 1.10\n'
                                            '    clat percentiles (nsec):\n'
                                            '     |  1.00th=[ 4384],  5.00th=[ '
                                            '4384], 10.00th=[ 4448], 20.00th=[ '
                                            '4448],\n'
                                            '     | 30.00th=[ 4448], 40.00th=[ '
                                            '4640], 50.00th=[ 4640], 60.00th=[ '
                                            '4640],\n'
                                            '     | 70.00th=[ 4704], 80.00th=[ '
                                            '4704], 90.00th=[ 4704], 95.00th=[ '
                                            '4896],\n'
                                            '     | 99.00th=[ 9664], '
                                            '99.50th=[10944], 99.90th=[15296], '
                                            '99.95th=[15424],\n'
                                            '     | 99.99th=[58624]\n'
                                            '   bw (  KiB/s): min=72756, '
                                            'max=73193, per=100.00%, '
                                            'avg=72972.93, stdev=114.13, '
                                            'samples=14\n'
                                            '   iops        : min=145512, '
                                            'max=146386, avg=145945.86, '
                                            'stdev=228.25, samples=14\n'
                                            '  lat (nsec)   : 750=0.01%, '
                                            '1000=0.01%\n'
                                            '  lat (usec)   : 2=0.01%, '
                                            '4=0.02%, 10=99.11%, 20=0.84%, '
                                            '50=0.01%\n'
                                            '  lat (usec)   : 100=0.01%, '
                                            '250=0.01%\n'
                                            '  cpu          : usr=54.73%, '
                                            'sys=45.24%, ctx=10, majf=0, '
                                            'minf=0\n'
                                            '  IO depths    : 1=100.0%, '
                                            '2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, >=64=0.0%\n'
                                            '     submit    : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     complete  : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     issued rwts: '
                                            'total=1021585,0,0,0 short=0,0,0,0 '
                                            'dropped=0,0,0,0\n'
                                            '     latency   : target=0, '
                                            'window=0, percentile=100.00%, '
                                            'depth=1\n'
                                            '\n'
                                            'Run status group 0 (all jobs):\n'
                                            '   READ: bw=71.2MiB/s (74.7MB/s), '
                                            '71.2MiB/s-71.2MiB/s '
                                            '(74.7MB/s-74.7MB/s), io=499MiB '
                                            '(523MB), run=7001-7001msec\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_84.output'),
                                  'state': {'begin': 1666782452.1262112,
                                            'cmd': 'cat '
                                                   '/tmp/fio-output_512_1_libaio-bdev_xnvme_0.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.7958531379699707,
                                            'end': 1666782452.9220643,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_84.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_84.state')},
                       'cmd_85': {'output': 'rm: cannot remove '
                                            "'/tmp/fio-output_512_1_libaio-bdev_xnvme_1.txt': "
                                            'No such file or directory\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_85.output'),
                                  'state': {'begin': 1666782455.0286593,
                                            'cmd': 'rm '
                                                   '/tmp/fio-output_512_1_libaio-bdev_xnvme_1.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.7907915115356445,
                                            'end': 1666782455.8194509,
                                            'err': 1,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_85.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_85.state')},
                       'cmd_86': {'output': 'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_86.output'),
                                  'state': {'begin': 1666782455.826383,
                                            'cmd': '/opt/fio/bin/fio '
                                                   '--name="libaio-bdev_xnvme" '
                                                   '--rw="randread" '
                                                   '--size="12G" --iodepth="1" '
                                                   '--bs="512" --direct="1" '
                                                   '--thread="1" '
                                                   '--time_based="1" '
                                                   '--runtime="7" '
                                                   '--ramp_time="3" '
                                                   '--norandommap="1" '
                                                   '--allow_file_create="0" '
                                                   '--ioengine="spdk_bdev" '
                                                   '--spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" '
                                                   '--filename="Nvme0n1" '
                                                   '--output-format="normal,json" '
                                                   '--output="/tmp/fio-output_512_1_libaio-bdev_xnvme_1.txt" '
                                                   '--cpus_allowed="1"',
                                            'cwd': 'None',
                                            'elapsed': 11.405316829681396,
                                            'end': 1666782467.2317,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_86.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_86.state')},
                       'cmd_87': {'output': 'libaio-bdev_xnvme: (g=0): '
                                            'rw=randread, bs=(R) 512B-512B, '
                                            '(W) 512B-512B, (T) 512B-512B, '
                                            'ioengine=spdk_bdev, iodepth=1\n'
                                            'fio-3.32-dirty\n'
                                            'Starting 1 thread\n'
                                            '{\n'
                                            '  "fio version" : '
                                            '"fio-3.32-dirty",\n'
                                            '  "timestamp" : 1666789557,\n'
                                            '  "timestamp_ms" : '
                                            '1666789557225,\n'
                                            '  "time" : "Wed Oct 26 15:05:57 '
                                            '2022",\n'
                                            '  "jobs" : [\n'
                                            '    {\n'
                                            '      "jobname" : '
                                            '"libaio-bdev_xnvme",\n'
                                            '      "groupid" : 0,\n'
                                            '      "error" : 0,\n'
                                            '      "eta" : 0,\n'
                                            '      "elapsed" : 11,\n'
                                            '      "job options" : {\n'
                                            '        "name" : '
                                            '"libaio-bdev_xnvme",\n'
                                            '        "rw" : "randread",\n'
                                            '        "size" : "12G",\n'
                                            '        "iodepth" : "1",\n'
                                            '        "bs" : "512",\n'
                                            '        "direct" : "1",\n'
                                            '        "thread" : "1",\n'
                                            '        "time_based" : "1",\n'
                                            '        "runtime" : "7",\n'
                                            '        "ramp_time" : "3",\n'
                                            '        "norandommap" : "1",\n'
                                            '        "allow_file_create" : '
                                            '"0",\n'
                                            '        "ioengine" : '
                                            '"spdk_bdev",\n'
                                            '        "spdk_conf" : '
                                            '"/tmp/spdk_bdev_xnvme_libaio.conf",\n'
                                            '        "filename" : "Nvme0n1",\n'
                                            '        "cpus_allowed" : "1"\n'
                                            '      },\n'
                                            '      "read" : {\n'
                                            '        "io_bytes" : 522268672,\n'
                                            '        "io_kbytes" : 510028,\n'
                                            '        "bw_bytes" : 74609810,\n'
                                            '        "bw" : 72861,\n'
                                            '        "iops" : 145722.142857,\n'
                                            '        "runtime" : 7000,\n'
                                            '        "total_ios" : 1020055,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 1773,\n'
                                            '          "max" : 14247,\n'
                                            '          "mean" : 1911.317889,\n'
                                            '          "stddev" : 104.278118,\n'
                                            '          "N" : 1020056\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 601,\n'
                                            '          "max" : 117801,\n'
                                            '          "mean" : 4724.615978,\n'
                                            '          "stddev" : '
                                            '1066.997954,\n'
                                            '          "N" : 1020055,\n'
                                            '          "percentile" : {\n'
                                            '            "1.000000" : 4256,\n'
                                            '            "5.000000" : 4448,\n'
                                            '            "10.000000" : 4448,\n'
                                            '            "20.000000" : 4448,\n'
                                            '            "30.000000" : 4512,\n'
                                            '            "40.000000" : 4512,\n'
                                            '            "50.000000" : 4640,\n'
                                            '            "60.000000" : 4704,\n'
                                            '            "70.000000" : 4704,\n'
                                            '            "80.000000" : 4704,\n'
                                            '            "90.000000" : 4768,\n'
                                            '            "95.000000" : 4768,\n'
                                            '            "99.000000" : 9664,\n'
                                            '            "99.500000" : 10944,\n'
                                            '            "99.900000" : 15296,\n'
                                            '            "99.950000" : 15424,\n'
                                            '            "99.990000" : 57088\n'
                                            '          }\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 5851,\n'
                                            '          "max" : 119655,\n'
                                            '          "mean" : 6635.933895,\n'
                                            '          "stddev" : '
                                            '1071.288496,\n'
                                            '          "N" : 1020055\n'
                                            '        },\n'
                                            '        "bw_min" : 72757,\n'
                                            '        "bw_max" : 73024,\n'
                                            '        "bw_agg" : 100.000000,\n'
                                            '        "bw_mean" : '
                                            '72863.785714,\n'
                                            '        "bw_dev" : 78.671839,\n'
                                            '        "bw_samples" : 14,\n'
                                            '        "iops_min" : 145514,\n'
                                            '        "iops_max" : 146048,\n'
                                            '        "iops_mean" : '
                                            '145727.642857,\n'
                                            '        "iops_stddev" : '
                                            '157.314779,\n'
                                            '        "iops_samples" : 14\n'
                                            '      },\n'
                                            '      "write" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "trim" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "sync" : {\n'
                                            '        "total_ios" : 0,\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        }\n'
                                            '      },\n'
                                            '      "job_runtime" : 7000,\n'
                                            '      "usr_cpu" : 48.328571,\n'
                                            '      "sys_cpu" : 51.642857,\n'
                                            '      "ctx" : 14,\n'
                                            '      "majf" : 0,\n'
                                            '      "minf" : 0,\n'
                                            '      "iodepth_level" : {\n'
                                            '        "1" : 100.000000,\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_submit" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_complete" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ns" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.010000,\n'
                                            '        "1000" : 0.010000\n'
                                            '      },\n'
                                            '      "latency_us" : {\n'
                                            '        "2" : 0.010000,\n'
                                            '        "4" : 0.014607,\n'
                                            '        "10" : 99.142791,\n'
                                            '        "20" : 0.821034,\n'
                                            '        "50" : 0.010000,\n'
                                            '        "100" : 0.012646,\n'
                                            '        "250" : 0.010000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ms" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000,\n'
                                            '        "2000" : 0.000000,\n'
                                            '        ">=2000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_depth" : 1,\n'
                                            '      "latency_target" : 0,\n'
                                            '      "latency_percentile" : '
                                            '100.000000,\n'
                                            '      "latency_window" : 0\n'
                                            '    }\n'
                                            '  ]\n'
                                            '}\n'
                                            '\n'
                                            'libaio-bdev_xnvme: (groupid=0, '
                                            'jobs=1): err= 0: pid=6400: Wed '
                                            'Oct 26 15:05:57 2022\n'
                                            '  read: IOPS=146k, BW=71.2MiB/s '
                                            '(74.6MB/s)(498MiB/7000msec)\n'
                                            '    slat (nsec): min=1773, '
                                            'max=14247, avg=1911.32, '
                                            'stdev=104.28\n'
                                            '    clat (nsec): min=601, '
                                            'max=117801, avg=4724.62, '
                                            'stdev=1067.00\n'
                                            '     lat (usec): min=5, max=119, '
                                            'avg= 6.64, stdev= 1.07\n'
                                            '    clat percentiles (nsec):\n'
                                            '     |  1.00th=[ 4256],  5.00th=[ '
                                            '4448], 10.00th=[ 4448], 20.00th=[ '
                                            '4448],\n'
                                            '     | 30.00th=[ 4512], 40.00th=[ '
                                            '4512], 50.00th=[ 4640], 60.00th=[ '
                                            '4704],\n'
                                            '     | 70.00th=[ 4704], 80.00th=[ '
                                            '4704], 90.00th=[ 4768], 95.00th=[ '
                                            '4768],\n'
                                            '     | 99.00th=[ 9664], '
                                            '99.50th=[10944], 99.90th=[15296], '
                                            '99.95th=[15424],\n'
                                            '     | 99.99th=[57088]\n'
                                            '   bw (  KiB/s): min=72757, '
                                            'max=73024, per=100.00%, '
                                            'avg=72863.79, stdev=78.67, '
                                            'samples=14\n'
                                            '   iops        : min=145514, '
                                            'max=146048, avg=145727.64, '
                                            'stdev=157.31, samples=14\n'
                                            '  lat (nsec)   : 750=0.01%, '
                                            '1000=0.01%\n'
                                            '  lat (usec)   : 2=0.01%, '
                                            '4=0.01%, 10=99.14%, 20=0.82%, '
                                            '50=0.01%\n'
                                            '  lat (usec)   : 100=0.01%, '
                                            '250=0.01%\n'
                                            '  cpu          : usr=48.33%, '
                                            'sys=51.64%, ctx=14, majf=0, '
                                            'minf=0\n'
                                            '  IO depths    : 1=100.0%, '
                                            '2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, >=64=0.0%\n'
                                            '     submit    : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     complete  : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     issued rwts: '
                                            'total=1020055,0,0,0 short=0,0,0,0 '
                                            'dropped=0,0,0,0\n'
                                            '     latency   : target=0, '
                                            'window=0, percentile=100.00%, '
                                            'depth=1\n'
                                            '\n'
                                            'Run status group 0 (all jobs):\n'
                                            '   READ: bw=71.2MiB/s (74.6MB/s), '
                                            '71.2MiB/s-71.2MiB/s '
                                            '(74.6MB/s-74.6MB/s), io=498MiB '
                                            '(522MB), run=7000-7000msec\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_87.output'),
                                  'state': {'begin': 1666782467.2342203,
                                            'cmd': 'cat '
                                                   '/tmp/fio-output_512_1_libaio-bdev_xnvme_1.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.8045120239257812,
                                            'end': 1666782468.0387323,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_87.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_87.state')},
                       'cmd_88': {'output': 'rm: cannot remove '
                                            "'/tmp/fio-output_512_1_libaio-bdev_xnvme_2.txt': "
                                            'No such file or directory\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_88.output'),
                                  'state': {'begin': 1666782470.2117727,
                                            'cmd': 'rm '
                                                   '/tmp/fio-output_512_1_libaio-bdev_xnvme_2.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.8008444309234619,
                                            'end': 1666782471.012617,
                                            'err': 1,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_88.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_88.state')},
                       'cmd_89': {'output': 'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_89.output'),
                                  'state': {'begin': 1666782471.0155253,
                                            'cmd': '/opt/fio/bin/fio '
                                                   '--name="libaio-bdev_xnvme" '
                                                   '--rw="randread" '
                                                   '--size="12G" --iodepth="1" '
                                                   '--bs="512" --direct="1" '
                                                   '--thread="1" '
                                                   '--time_based="1" '
                                                   '--runtime="7" '
                                                   '--ramp_time="3" '
                                                   '--norandommap="1" '
                                                   '--allow_file_create="0" '
                                                   '--ioengine="spdk_bdev" '
                                                   '--spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" '
                                                   '--filename="Nvme0n1" '
                                                   '--output-format="normal,json" '
                                                   '--output="/tmp/fio-output_512_1_libaio-bdev_xnvme_2.txt" '
                                                   '--cpus_allowed="1"',
                                            'cwd': 'None',
                                            'elapsed': 11.44051480293274,
                                            'end': 1666782482.4560401,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_89.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_89.state')},
                       'cmd_90': {'output': 'libaio-bdev_xnvme: (g=0): '
                                            'rw=randread, bs=(R) 512B-512B, '
                                            '(W) 512B-512B, (T) 512B-512B, '
                                            'ioengine=spdk_bdev, iodepth=1\n'
                                            'fio-3.32-dirty\n'
                                            'Starting 1 thread\n'
                                            '{\n'
                                            '  "fio version" : '
                                            '"fio-3.32-dirty",\n'
                                            '  "timestamp" : 1666789572,\n'
                                            '  "timestamp_ms" : '
                                            '1666789572451,\n'
                                            '  "time" : "Wed Oct 26 15:06:12 '
                                            '2022",\n'
                                            '  "jobs" : [\n'
                                            '    {\n'
                                            '      "jobname" : '
                                            '"libaio-bdev_xnvme",\n'
                                            '      "groupid" : 0,\n'
                                            '      "error" : 0,\n'
                                            '      "eta" : 0,\n'
                                            '      "elapsed" : 11,\n'
                                            '      "job options" : {\n'
                                            '        "name" : '
                                            '"libaio-bdev_xnvme",\n'
                                            '        "rw" : "randread",\n'
                                            '        "size" : "12G",\n'
                                            '        "iodepth" : "1",\n'
                                            '        "bs" : "512",\n'
                                            '        "direct" : "1",\n'
                                            '        "thread" : "1",\n'
                                            '        "time_based" : "1",\n'
                                            '        "runtime" : "7",\n'
                                            '        "ramp_time" : "3",\n'
                                            '        "norandommap" : "1",\n'
                                            '        "allow_file_create" : '
                                            '"0",\n'
                                            '        "ioengine" : '
                                            '"spdk_bdev",\n'
                                            '        "spdk_conf" : '
                                            '"/tmp/spdk_bdev_xnvme_libaio.conf",\n'
                                            '        "filename" : "Nvme0n1",\n'
                                            '        "cpus_allowed" : "1"\n'
                                            '      },\n'
                                            '      "read" : {\n'
                                            '        "io_bytes" : 521175040,\n'
                                            '        "io_kbytes" : 508960,\n'
                                            '        "bw_bytes" : 74442942,\n'
                                            '        "bw" : 72698,\n'
                                            '        "iops" : 145396.371947,\n'
                                            '        "runtime" : 7001,\n'
                                            '        "total_ios" : 1017920,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 1743,\n'
                                            '          "max" : 14618,\n'
                                            '          "mean" : 1896.726667,\n'
                                            '          "stddev" : 103.752460,\n'
                                            '          "N" : 1017920\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 601,\n'
                                            '          "max" : 117551,\n'
                                            '          "mean" : 4735.870113,\n'
                                            '          "stddev" : '
                                            '1060.455946,\n'
                                            '          "N" : 1017920,\n'
                                            '          "percentile" : {\n'
                                            '            "1.000000" : 4384,\n'
                                            '            "5.000000" : 4384,\n'
                                            '            "10.000000" : 4448,\n'
                                            '            "20.000000" : 4448,\n'
                                            '            "30.000000" : 4640,\n'
                                            '            "40.000000" : 4640,\n'
                                            '            "50.000000" : 4640,\n'
                                            '            "60.000000" : 4640,\n'
                                            '            "70.000000" : 4704,\n'
                                            '            "80.000000" : 4704,\n'
                                            '            "90.000000" : 4704,\n'
                                            '            "95.000000" : 4896,\n'
                                            '            "99.000000" : 9792,\n'
                                            '            "99.500000" : 10816,\n'
                                            '            "99.900000" : 15168,\n'
                                            '            "99.950000" : 15424,\n'
                                            '            "99.990000" : 56576\n'
                                            '          }\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 5891,\n'
                                            '          "max" : 119544,\n'
                                            '          "mean" : 6632.596780,\n'
                                            '          "stddev" : '
                                            '1064.048858,\n'
                                            '          "N" : 1017920\n'
                                            '        },\n'
                                            '        "bw_min" : 72585,\n'
                                            '        "bw_max" : 72865,\n'
                                            '        "bw_agg" : 100.000000,\n'
                                            '        "bw_mean" : '
                                            '72711.142857,\n'
                                            '        "bw_dev" : 64.441813,\n'
                                            '        "bw_samples" : 14,\n'
                                            '        "iops_min" : 145170,\n'
                                            '        "iops_max" : 145730,\n'
                                            '        "iops_mean" : '
                                            '145422.285714,\n'
                                            '        "iops_stddev" : '
                                            '128.883626,\n'
                                            '        "iops_samples" : 14\n'
                                            '      },\n'
                                            '      "write" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "trim" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "sync" : {\n'
                                            '        "total_ios" : 0,\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        }\n'
                                            '      },\n'
                                            '      "job_runtime" : 7000,\n'
                                            '      "usr_cpu" : 53.528571,\n'
                                            '      "sys_cpu" : 46.457143,\n'
                                            '      "ctx" : 10,\n'
                                            '      "majf" : 0,\n'
                                            '      "minf" : 0,\n'
                                            '      "iodepth_level" : {\n'
                                            '        "1" : 100.000000,\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_submit" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_complete" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ns" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.010000,\n'
                                            '        "1000" : 0.010000\n'
                                            '      },\n'
                                            '      "latency_us" : {\n'
                                            '        "2" : 0.010000,\n'
                                            '        "4" : 0.016111,\n'
                                            '        "10" : 99.099733,\n'
                                            '        "20" : 0.863329,\n'
                                            '        "50" : 0.010000,\n'
                                            '        "100" : 0.011199,\n'
                                            '        "250" : 0.010000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ms" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000,\n'
                                            '        "2000" : 0.000000,\n'
                                            '        ">=2000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_depth" : 1,\n'
                                            '      "latency_target" : 0,\n'
                                            '      "latency_percentile" : '
                                            '100.000000,\n'
                                            '      "latency_window" : 0\n'
                                            '    }\n'
                                            '  ]\n'
                                            '}\n'
                                            '\n'
                                            'libaio-bdev_xnvme: (groupid=0, '
                                            'jobs=1): err= 0: pid=6458: Wed '
                                            'Oct 26 15:06:12 2022\n'
                                            '  read: IOPS=145k, BW=71.0MiB/s '
                                            '(74.4MB/s)(497MiB/7001msec)\n'
                                            '    slat (nsec): min=1743, '
                                            'max=14618, avg=1896.73, '
                                            'stdev=103.75\n'
                                            '    clat (nsec): min=601, '
                                            'max=117551, avg=4735.87, '
                                            'stdev=1060.46\n'
                                            '     lat (usec): min=5, max=119, '
                                            'avg= 6.63, stdev= 1.06\n'
                                            '    clat percentiles (nsec):\n'
                                            '     |  1.00th=[ 4384],  5.00th=[ '
                                            '4384], 10.00th=[ 4448], 20.00th=[ '
                                            '4448],\n'
                                            '     | 30.00th=[ 4640], 40.00th=[ '
                                            '4640], 50.00th=[ 4640], 60.00th=[ '
                                            '4640],\n'
                                            '     | 70.00th=[ 4704], 80.00th=[ '
                                            '4704], 90.00th=[ 4704], 95.00th=[ '
                                            '4896],\n'
                                            '     | 99.00th=[ 9792], '
                                            '99.50th=[10816], 99.90th=[15168], '
                                            '99.95th=[15424],\n'
                                            '     | 99.99th=[56576]\n'
                                            '   bw (  KiB/s): min=72585, '
                                            'max=72865, per=100.00%, '
                                            'avg=72711.14, stdev=64.44, '
                                            'samples=14\n'
                                            '   iops        : min=145170, '
                                            'max=145730, avg=145422.29, '
                                            'stdev=128.88, samples=14\n'
                                            '  lat (nsec)   : 750=0.01%, '
                                            '1000=0.01%\n'
                                            '  lat (usec)   : 2=0.01%, '
                                            '4=0.02%, 10=99.10%, 20=0.86%, '
                                            '50=0.01%\n'
                                            '  lat (usec)   : 100=0.01%, '
                                            '250=0.01%\n'
                                            '  cpu          : usr=53.53%, '
                                            'sys=46.46%, ctx=10, majf=0, '
                                            'minf=0\n'
                                            '  IO depths    : 1=100.0%, '
                                            '2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, >=64=0.0%\n'
                                            '     submit    : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     complete  : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     issued rwts: '
                                            'total=1017920,0,0,0 short=0,0,0,0 '
                                            'dropped=0,0,0,0\n'
                                            '     latency   : target=0, '
                                            'window=0, percentile=100.00%, '
                                            'depth=1\n'
                                            '\n'
                                            'Run status group 0 (all jobs):\n'
                                            '   READ: bw=71.0MiB/s (74.4MB/s), '
                                            '71.0MiB/s-71.0MiB/s '
                                            '(74.4MB/s-74.4MB/s), io=497MiB '
                                            '(521MB), run=7001-7001msec\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_90.output'),
                                  'state': {'begin': 1666782482.4585795,
                                            'cmd': 'cat '
                                                   '/tmp/fio-output_512_1_libaio-bdev_xnvme_2.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.7996425628662109,
                                            'end': 1666782483.258222,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_90.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_90.state')},
                       'cmd_91': {'output': 'rm: cannot remove '
                                            "'/tmp/fio-output_512_1_libaio-xnvme_0.txt': "
                                            'No such file or directory\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_91.output'),
                                  'state': {'begin': 1666782484.3010526,
                                            'cmd': 'rm '
                                                   '/tmp/fio-output_512_1_libaio-xnvme_0.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.7843501567840576,
                                            'end': 1666782485.0854027,
                                            'err': 1,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_91.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_91.state')},
                       'cmd_92': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_92.output'),
                                  'state': {'begin': 1666782485.0873108,
                                            'cmd': '/opt/fio/bin/fio '
                                                   '--name="libaio-xnvme" '
                                                   '--rw="randread" '
                                                   '--size="12G" --iodepth="1" '
                                                   '--bs="512" --direct="1" '
                                                   '--thread="1" '
                                                   '--time_based="1" '
                                                   '--runtime="7" '
                                                   '--ramp_time="3" '
                                                   '--norandommap="1" '
                                                   '--allow_file_create="0" '
                                                   '--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
                                                   '--xnvme_be="linux" '
                                                   '--xnvme_async="libaio" '
                                                   '--xnvme_sync="nvme" '
                                                   '--xnvme_admin="nvme" '
                                                   '--xnvme_dev_nsid="1" '
                                                   '--filename="/dev/nvme0n1" '
                                                   '--output-format="normal,json" '
                                                   '--output="/tmp/fio-output_512_1_libaio-xnvme_0.txt" '
                                                   '--cpus_allowed="1"',
                                            'cwd': 'None',
                                            'elapsed': 11.084620952606201,
                                            'end': 1666782496.1719317,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_92.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_92.state')},
                       'cmd_93': {'output': 'libaio-xnvme: (g=0): rw=randread, '
                                            'bs=(R) 512B-512B, (W) 512B-512B, '
                                            '(T) 512B-512B, ioengine=xnvme, '
                                            'iodepth=1\n'
                                            'fio-3.32-dirty\n'
                                            'Starting 1 thread\n'
                                            '{\n'
                                            '  "fio version" : '
                                            '"fio-3.32-dirty",\n'
                                            '  "timestamp" : 1666789586,\n'
                                            '  "timestamp_ms" : '
                                            '1666789586262,\n'
                                            '  "time" : "Wed Oct 26 15:06:26 '
                                            '2022",\n'
                                            '  "jobs" : [\n'
                                            '    {\n'
                                            '      "jobname" : '
                                            '"libaio-xnvme",\n'
                                            '      "groupid" : 0,\n'
                                            '      "error" : 0,\n'
                                            '      "eta" : 0,\n'
                                            '      "elapsed" : 11,\n'
                                            '      "job options" : {\n'
                                            '        "name" : "libaio-xnvme",\n'
                                            '        "rw" : "randread",\n'
                                            '        "size" : "12G",\n'
                                            '        "iodepth" : "1",\n'
                                            '        "bs" : "512",\n'
                                            '        "direct" : "1",\n'
                                            '        "thread" : "1",\n'
                                            '        "time_based" : "1",\n'
                                            '        "runtime" : "7",\n'
                                            '        "ramp_time" : "3",\n'
                                            '        "norandommap" : "1",\n'
                                            '        "allow_file_create" : '
                                            '"0",\n'
                                            '        "ioengine" : '
                                            '"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
                                            '        "xnvme_be" : "linux",\n'
                                            '        "xnvme_async" : '
                                            '"libaio",\n'
                                            '        "xnvme_sync" : "nvme",\n'
                                            '        "xnvme_admin" : "nvme",\n'
                                            '        "xnvme_dev_nsid" : "1",\n'
                                            '        "filename" : '
                                            '"/dev/nvme0n1",\n'
                                            '        "cpus_allowed" : "1"\n'
                                            '      },\n'
                                            '      "read" : {\n'
                                            '        "io_bytes" : 384857088,\n'
                                            '        "io_kbytes" : 375837,\n'
                                            '        "bw_bytes" : 54979584,\n'
                                            '        "bw" : 53691,\n'
                                            '        "iops" : 107381.857143,\n'
                                            '        "runtime" : 7000,\n'
                                            '        "total_ios" : 751673,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 1693,\n'
                                            '          "max" : 46567,\n'
                                            '          "mean" : 1810.359939,\n'
                                            '          "stddev" : 98.872126,\n'
                                            '          "N" : 751674\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 612,\n'
                                            '          "max" : 75983,\n'
                                            '          "mean" : 7221.342058,\n'
                                            '          "stddev" : 966.759816,\n'
                                            '          "N" : 751673,\n'
                                            '          "percentile" : {\n'
                                            '            "1.000000" : 6752,\n'
                                            '            "5.000000" : 6944,\n'
                                            '            "10.000000" : 7008,\n'
                                            '            "20.000000" : 7008,\n'
                                            '            "30.000000" : 7072,\n'
                                            '            "40.000000" : 7072,\n'
                                            '            "50.000000" : 7136,\n'
                                            '            "60.000000" : 7200,\n'
                                            '            "70.000000" : 7200,\n'
                                            '            "80.000000" : 7200,\n'
                                            '            "90.000000" : 7264,\n'
                                            '            "95.000000" : 7392,\n'
                                            '            "99.000000" : 10048,\n'
                                            '            "99.500000" : 13632,\n'
                                            '            "99.900000" : 15424,\n'
                                            '            "99.950000" : 15808,\n'
                                            '            "99.990000" : 57600\n'
                                            '          }\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 7604,\n'
                                            '          "max" : 77766,\n'
                                            '          "mean" : 9031.702033,\n'
                                            '          "stddev" : 973.215408,\n'
                                            '          "N" : 751673\n'
                                            '        },\n'
                                            '        "bw_min" : 53585,\n'
                                            '        "bw_max" : 53789,\n'
                                            '        "bw_agg" : 100.000000,\n'
                                            '        "bw_mean" : '
                                            '53692.785714,\n'
                                            '        "bw_dev" : 71.730455,\n'
                                            '        "bw_samples" : 14,\n'
                                            '        "iops_min" : 107170,\n'
                                            '        "iops_max" : 107578,\n'
                                            '        "iops_mean" : '
                                            '107385.642857,\n'
                                            '        "iops_stddev" : '
                                            '143.549298,\n'
                                            '        "iops_samples" : 14\n'
                                            '      },\n'
                                            '      "write" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "trim" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "sync" : {\n'
                                            '        "total_ios" : 0,\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        }\n'
                                            '      },\n'
                                            '      "job_runtime" : 7000,\n'
                                            '      "usr_cpu" : 11.500000,\n'
                                            '      "sys_cpu" : 35.457143,\n'
                                            '      "ctx" : 751676,\n'
                                            '      "majf" : 0,\n'
                                            '      "minf" : 0,\n'
                                            '      "iodepth_level" : {\n'
                                            '        "1" : 100.000000,\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_submit" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_complete" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ns" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.010000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_us" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.010000,\n'
                                            '        "10" : 98.988124,\n'
                                            '        "20" : 0.992320,\n'
                                            '        "50" : 0.010000,\n'
                                            '        "100" : 0.014767,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ms" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000,\n'
                                            '        "2000" : 0.000000,\n'
                                            '        ">=2000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_depth" : 1,\n'
                                            '      "latency_target" : 0,\n'
                                            '      "latency_percentile" : '
                                            '100.000000,\n'
                                            '      "latency_window" : 0\n'
                                            '    }\n'
                                            '  ]\n'
                                            '}\n'
                                            '\n'
                                            'libaio-xnvme: (groupid=0, '
                                            'jobs=1): err= 0: pid=6506: Wed '
                                            'Oct 26 15:06:26 2022\n'
                                            '  read: IOPS=107k, BW=52.4MiB/s '
                                            '(55.0MB/s)(367MiB/7000msec)\n'
                                            '    slat (nsec): min=1693, '
                                            'max=46567, avg=1810.36, '
                                            'stdev=98.87\n'
                                            '    clat (nsec): min=612, '
                                            'max=75983, avg=7221.34, '
                                            'stdev=966.76\n'
                                            '     lat (nsec): min=7604, '
                                            'max=77766, avg=9031.70, '
                                            'stdev=973.22\n'
                                            '    clat percentiles (nsec):\n'
                                            '     |  1.00th=[ 6752],  5.00th=[ '
                                            '6944], 10.00th=[ 7008], 20.00th=[ '
                                            '7008],\n'
                                            '     | 30.00th=[ 7072], 40.00th=[ '
                                            '7072], 50.00th=[ 7136], 60.00th=[ '
                                            '7200],\n'
                                            '     | 70.00th=[ 7200], 80.00th=[ '
                                            '7200], 90.00th=[ 7264], 95.00th=[ '
                                            '7392],\n'
                                            '     | 99.00th=[10048], '
                                            '99.50th=[13632], 99.90th=[15424], '
                                            '99.95th=[15808],\n'
                                            '     | 99.99th=[57600]\n'
                                            '   bw (  KiB/s): min=53585, '
                                            'max=53789, per=100.00%, '
                                            'avg=53692.79, stdev=71.73, '
                                            'samples=14\n'
                                            '   iops        : min=107170, '
                                            'max=107578, avg=107385.64, '
                                            'stdev=143.55, samples=14\n'
                                            '  lat (nsec)   : 750=0.01%\n'
                                            '  lat (usec)   : 4=0.01%, '
                                            '10=98.99%, 20=0.99%, 50=0.01%, '
                                            '100=0.01%\n'
                                            '  cpu          : usr=11.50%, '
                                            'sys=35.46%, ctx=751676, majf=0, '
                                            'minf=0\n'
                                            '  IO depths    : 1=100.0%, '
                                            '2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, >=64=0.0%\n'
                                            '     submit    : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     complete  : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     issued rwts: '
                                            'total=751673,0,0,0 short=0,0,0,0 '
                                            'dropped=0,0,0,0\n'
                                            '     latency   : target=0, '
                                            'window=0, percentile=100.00%, '
                                            'depth=1\n'
                                            '\n'
                                            'Run status group 0 (all jobs):\n'
                                            '   READ: bw=52.4MiB/s (55.0MB/s), '
                                            '52.4MiB/s-52.4MiB/s '
                                            '(55.0MB/s-55.0MB/s), io=367MiB '
                                            '(385MB), run=7000-7000msec\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_93.output'),
                                  'state': {'begin': 1666782496.1747887,
                                            'cmd': 'cat '
                                                   '/tmp/fio-output_512_1_libaio-xnvme_0.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.8476107120513916,
                                            'end': 1666782497.0223994,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_93.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_93.state')},
                       'cmd_94': {'output': 'rm: cannot remove '
                                            "'/tmp/fio-output_512_1_libaio-xnvme_1.txt': "
                                            'No such file or directory\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_94.output'),
                                  'state': {'begin': 1666782498.1266768,
                                            'cmd': 'rm '
                                                   '/tmp/fio-output_512_1_libaio-xnvme_1.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.8207857608795166,
                                            'end': 1666782498.9474626,
                                            'err': 1,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_94.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_94.state')},
                       'cmd_95': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_95.output'),
                                  'state': {'begin': 1666782498.949676,
                                            'cmd': '/opt/fio/bin/fio '
                                                   '--name="libaio-xnvme" '
                                                   '--rw="randread" '
                                                   '--size="12G" --iodepth="1" '
                                                   '--bs="512" --direct="1" '
                                                   '--thread="1" '
                                                   '--time_based="1" '
                                                   '--runtime="7" '
                                                   '--ramp_time="3" '
                                                   '--norandommap="1" '
                                                   '--allow_file_create="0" '
                                                   '--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
                                                   '--xnvme_be="linux" '
                                                   '--xnvme_async="libaio" '
                                                   '--xnvme_sync="nvme" '
                                                   '--xnvme_admin="nvme" '
                                                   '--xnvme_dev_nsid="1" '
                                                   '--filename="/dev/nvme0n1" '
                                                   '--output-format="normal,json" '
                                                   '--output="/tmp/fio-output_512_1_libaio-xnvme_1.txt" '
                                                   '--cpus_allowed="1"',
                                            'cwd': 'None',
                                            'elapsed': 11.12022852897644,
                                            'end': 1666782510.0699046,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_95.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_95.state')},
                       'cmd_96': {'output': 'libaio-xnvme: (g=0): rw=randread, '
                                            'bs=(R) 512B-512B, (W) 512B-512B, '
                                            '(T) 512B-512B, ioengine=xnvme, '
                                            'iodepth=1\n'
                                            'fio-3.32-dirty\n'
                                            'Starting 1 thread\n'
                                            '{\n'
                                            '  "fio version" : '
                                            '"fio-3.32-dirty",\n'
                                            '  "timestamp" : 1666789600,\n'
                                            '  "timestamp_ms" : '
                                            '1666789600159,\n'
                                            '  "time" : "Wed Oct 26 15:06:40 '
                                            '2022",\n'
                                            '  "jobs" : [\n'
                                            '    {\n'
                                            '      "jobname" : '
                                            '"libaio-xnvme",\n'
                                            '      "groupid" : 0,\n'
                                            '      "error" : 0,\n'
                                            '      "eta" : 0,\n'
                                            '      "elapsed" : 11,\n'
                                            '      "job options" : {\n'
                                            '        "name" : "libaio-xnvme",\n'
                                            '        "rw" : "randread",\n'
                                            '        "size" : "12G",\n'
                                            '        "iodepth" : "1",\n'
                                            '        "bs" : "512",\n'
                                            '        "direct" : "1",\n'
                                            '        "thread" : "1",\n'
                                            '        "time_based" : "1",\n'
                                            '        "runtime" : "7",\n'
                                            '        "ramp_time" : "3",\n'
                                            '        "norandommap" : "1",\n'
                                            '        "allow_file_create" : '
                                            '"0",\n'
                                            '        "ioengine" : '
                                            '"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
                                            '        "xnvme_be" : "linux",\n'
                                            '        "xnvme_async" : '
                                            '"libaio",\n'
                                            '        "xnvme_sync" : "nvme",\n'
                                            '        "xnvme_admin" : "nvme",\n'
                                            '        "xnvme_dev_nsid" : "1",\n'
                                            '        "filename" : '
                                            '"/dev/nvme0n1",\n'
                                            '        "cpus_allowed" : "1"\n'
                                            '      },\n'
                                            '      "read" : {\n'
                                            '        "io_bytes" : 385519104,\n'
                                            '        "io_kbytes" : 376483,\n'
                                            '        "bw_bytes" : 55066291,\n'
                                            '        "bw" : 53775,\n'
                                            '        "iops" : 107551.349807,\n'
                                            '        "runtime" : 7001,\n'
                                            '        "total_ios" : 752967,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 1693,\n'
                                            '          "max" : 22682,\n'
                                            '          "mean" : 1809.645753,\n'
                                            '          "stddev" : 88.186978,\n'
                                            '          "N" : 752967\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 672,\n'
                                            '          "max" : 75631,\n'
                                            '          "mean" : 7207.420906,\n'
                                            '          "stddev" : '
                                            '1021.305121,\n'
                                            '          "N" : 752967,\n'
                                            '          "percentile" : {\n'
                                            '            "1.000000" : 6752,\n'
                                            '            "5.000000" : 6944,\n'
                                            '            "10.000000" : 6944,\n'
                                            '            "20.000000" : 7008,\n'
                                            '            "30.000000" : 7008,\n'
                                            '            "40.000000" : 7072,\n'
                                            '            "50.000000" : 7136,\n'
                                            '            "60.000000" : 7136,\n'
                                            '            "70.000000" : 7200,\n'
                                            '            "80.000000" : 7200,\n'
                                            '            "90.000000" : 7264,\n'
                                            '            "95.000000" : 7392,\n'
                                            '            "99.000000" : 10048,\n'
                                            '            "99.500000" : 13504,\n'
                                            '            "99.900000" : 15424,\n'
                                            '            "99.950000" : 15808,\n'
                                            '            "99.990000" : 60160\n'
                                            '          }\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 7604,\n'
                                            '          "max" : 77385,\n'
                                            '          "mean" : 9017.066659,\n'
                                            '          "stddev" : '
                                            '1026.795108,\n'
                                            '          "N" : 752967\n'
                                            '        },\n'
                                            '        "bw_min" : 53671,\n'
                                            '        "bw_max" : 53904,\n'
                                            '        "bw_agg" : 100.000000,\n'
                                            '        "bw_mean" : '
                                            '53804.928571,\n'
                                            '        "bw_dev" : 70.010400,\n'
                                            '        "bw_samples" : 14,\n'
                                            '        "iops_min" : 107342,\n'
                                            '        "iops_max" : 107808,\n'
                                            '        "iops_mean" : '
                                            '107610.000000,\n'
                                            '        "iops_stddev" : '
                                            '139.982966,\n'
                                            '        "iops_samples" : 14\n'
                                            '      },\n'
                                            '      "write" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "trim" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "sync" : {\n'
                                            '        "total_ios" : 0,\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        }\n'
                                            '      },\n'
                                            '      "job_runtime" : 7000,\n'
                                            '      "usr_cpu" : 11.300000,\n'
                                            '      "sys_cpu" : 35.500000,\n'
                                            '      "ctx" : 752967,\n'
                                            '      "majf" : 0,\n'
                                            '      "minf" : 0,\n'
                                            '      "iodepth_level" : {\n'
                                            '        "1" : 100.000000,\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_submit" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_complete" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ns" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.010000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_us" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.010000,\n'
                                            '        "10" : 98.972863,\n'
                                            '        "20" : 1.005755,\n'
                                            '        "50" : 0.010000,\n'
                                            '        "100" : 0.017265,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ms" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000,\n'
                                            '        "2000" : 0.000000,\n'
                                            '        ">=2000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_depth" : 1,\n'
                                            '      "latency_target" : 0,\n'
                                            '      "latency_percentile" : '
                                            '100.000000,\n'
                                            '      "latency_window" : 0\n'
                                            '    }\n'
                                            '  ]\n'
                                            '}\n'
                                            '\n'
                                            'libaio-xnvme: (groupid=0, '
                                            'jobs=1): err= 0: pid=6554: Wed '
                                            'Oct 26 15:06:40 2022\n'
                                            '  read: IOPS=108k, BW=52.5MiB/s '
                                            '(55.1MB/s)(368MiB/7001msec)\n'
                                            '    slat (nsec): min=1693, '
                                            'max=22682, avg=1809.65, '
                                            'stdev=88.19\n'
                                            '    clat (nsec): min=672, '
                                            'max=75631, avg=7207.42, '
                                            'stdev=1021.31\n'
                                            '     lat (nsec): min=7604, '
                                            'max=77385, avg=9017.07, '
                                            'stdev=1026.80\n'
                                            '    clat percentiles (nsec):\n'
                                            '     |  1.00th=[ 6752],  5.00th=[ '
                                            '6944], 10.00th=[ 6944], 20.00th=[ '
                                            '7008],\n'
                                            '     | 30.00th=[ 7008], 40.00th=[ '
                                            '7072], 50.00th=[ 7136], 60.00th=[ '
                                            '7136],\n'
                                            '     | 70.00th=[ 7200], 80.00th=[ '
                                            '7200], 90.00th=[ 7264], 95.00th=[ '
                                            '7392],\n'
                                            '     | 99.00th=[10048], '
                                            '99.50th=[13504], 99.90th=[15424], '
                                            '99.95th=[15808],\n'
                                            '     | 99.99th=[60160]\n'
                                            '   bw (  KiB/s): min=53671, '
                                            'max=53904, per=100.00%, '
                                            'avg=53804.93, stdev=70.01, '
                                            'samples=14\n'
                                            '   iops        : min=107342, '
                                            'max=107808, avg=107610.00, '
                                            'stdev=139.98, samples=14\n'
                                            '  lat (nsec)   : 750=0.01%\n'
                                            '  lat (usec)   : 4=0.01%, '
                                            '10=98.97%, 20=1.01%, 50=0.01%, '
                                            '100=0.02%\n'
                                            '  cpu          : usr=11.30%, '
                                            'sys=35.50%, ctx=752967, majf=0, '
                                            'minf=0\n'
                                            '  IO depths    : 1=100.0%, '
                                            '2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, >=64=0.0%\n'
                                            '     submit    : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     complete  : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     issued rwts: '
                                            'total=752967,0,0,0 short=0,0,0,0 '
                                            'dropped=0,0,0,0\n'
                                            '     latency   : target=0, '
                                            'window=0, percentile=100.00%, '
                                            'depth=1\n'
                                            '\n'
                                            'Run status group 0 (all jobs):\n'
                                            '   READ: bw=52.5MiB/s (55.1MB/s), '
                                            '52.5MiB/s-52.5MiB/s '
                                            '(55.1MB/s-55.1MB/s), io=368MiB '
                                            '(386MB), run=7001-7001msec\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_96.output'),
                                  'state': {'begin': 1666782510.072631,
                                            'cmd': 'cat '
                                                   '/tmp/fio-output_512_1_libaio-xnvme_1.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.8313148021697998,
                                            'end': 1666782510.9039457,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_96.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_96.state')},
                       'cmd_97': {'output': 'rm: cannot remove '
                                            "'/tmp/fio-output_512_1_libaio-xnvme_2.txt': "
                                            'No such file or directory\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_97.output'),
                                  'state': {'begin': 1666782512.010404,
                                            'cmd': 'rm '
                                                   '/tmp/fio-output_512_1_libaio-xnvme_2.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.8171346187591553,
                                            'end': 1666782512.8275387,
                                            'err': 1,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_97.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_97.state')},
                       'cmd_98': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_98.output'),
                                  'state': {'begin': 1666782512.8297102,
                                            'cmd': '/opt/fio/bin/fio '
                                                   '--name="libaio-xnvme" '
                                                   '--rw="randread" '
                                                   '--size="12G" --iodepth="1" '
                                                   '--bs="512" --direct="1" '
                                                   '--thread="1" '
                                                   '--time_based="1" '
                                                   '--runtime="7" '
                                                   '--ramp_time="3" '
                                                   '--norandommap="1" '
                                                   '--allow_file_create="0" '
                                                   '--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
                                                   '--xnvme_be="linux" '
                                                   '--xnvme_async="libaio" '
                                                   '--xnvme_sync="nvme" '
                                                   '--xnvme_admin="nvme" '
                                                   '--xnvme_dev_nsid="1" '
                                                   '--filename="/dev/nvme0n1" '
                                                   '--output-format="normal,json" '
                                                   '--output="/tmp/fio-output_512_1_libaio-xnvme_2.txt" '
                                                   '--cpus_allowed="1"',
                                            'cwd': 'None',
                                            'elapsed': 11.117222547531128,
                                            'end': 1666782523.9469328,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_98.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_98.state')},
                       'cmd_99': {'output': 'libaio-xnvme: (g=0): rw=randread, '
                                            'bs=(R) 512B-512B, (W) 512B-512B, '
                                            '(T) 512B-512B, ioengine=xnvme, '
                                            'iodepth=1\n'
                                            'fio-3.32-dirty\n'
                                            'Starting 1 thread\n'
                                            '{\n'
                                            '  "fio version" : '
                                            '"fio-3.32-dirty",\n'
                                            '  "timestamp" : 1666789614,\n'
                                            '  "timestamp_ms" : '
                                            '1666789614006,\n'
                                            '  "time" : "Wed Oct 26 15:06:54 '
                                            '2022",\n'
                                            '  "jobs" : [\n'
                                            '    {\n'
                                            '      "jobname" : '
                                            '"libaio-xnvme",\n'
                                            '      "groupid" : 0,\n'
                                            '      "error" : 0,\n'
                                            '      "eta" : 0,\n'
                                            '      "elapsed" : 11,\n'
                                            '      "job options" : {\n'
                                            '        "name" : "libaio-xnvme",\n'
                                            '        "rw" : "randread",\n'
                                            '        "size" : "12G",\n'
                                            '        "iodepth" : "1",\n'
                                            '        "bs" : "512",\n'
                                            '        "direct" : "1",\n'
                                            '        "thread" : "1",\n'
                                            '        "time_based" : "1",\n'
                                            '        "runtime" : "7",\n'
                                            '        "ramp_time" : "3",\n'
                                            '        "norandommap" : "1",\n'
                                            '        "allow_file_create" : '
                                            '"0",\n'
                                            '        "ioengine" : '
                                            '"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
                                            '        "xnvme_be" : "linux",\n'
                                            '        "xnvme_async" : '
                                            '"libaio",\n'
                                            '        "xnvme_sync" : "nvme",\n'
                                            '        "xnvme_admin" : "nvme",\n'
                                            '        "xnvme_dev_nsid" : "1",\n'
                                            '        "filename" : '
                                            '"/dev/nvme0n1",\n'
                                            '        "cpus_allowed" : "1"\n'
                                            '      },\n'
                                            '      "read" : {\n'
                                            '        "io_bytes" : 382008832,\n'
                                            '        "io_kbytes" : 373055,\n'
                                            '        "bw_bytes" : 54564895,\n'
                                            '        "bw" : 53286,\n'
                                            '        "iops" : 106571.918297,\n'
                                            '        "runtime" : 7001,\n'
                                            '        "total_ios" : 746110,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 1743,\n'
                                            '          "max" : 20578,\n'
                                            '          "mean" : 1853.316921,\n'
                                            '          "stddev" : 84.943754,\n'
                                            '          "N" : 746111\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 571,\n'
                                            '          "max" : 462377,\n'
                                            '          "mean" : 7237.205919,\n'
                                            '          "stddev" : '
                                            '1091.433101,\n'
                                            '          "N" : 746110,\n'
                                            '          "percentile" : {\n'
                                            '            "1.000000" : 6816,\n'
                                            '            "5.000000" : 6944,\n'
                                            '            "10.000000" : 7008,\n'
                                            '            "20.000000" : 7072,\n'
                                            '            "30.000000" : 7072,\n'
                                            '            "40.000000" : 7072,\n'
                                            '            "50.000000" : 7136,\n'
                                            '            "60.000000" : 7200,\n'
                                            '            "70.000000" : 7200,\n'
                                            '            "80.000000" : 7264,\n'
                                            '            "90.000000" : 7328,\n'
                                            '            "95.000000" : 7456,\n'
                                            '            "99.000000" : 9920,\n'
                                            '            "99.500000" : 13632,\n'
                                            '            "99.900000" : 15424,\n'
                                            '            "99.950000" : 16064,\n'
                                            '            "99.990000" : 57600\n'
                                            '          }\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 6503,\n'
                                            '          "max" : 464231,\n'
                                            '          "mean" : 9090.522852,\n'
                                            '          "stddev" : '
                                            '1096.371180,\n'
                                            '          "N" : 746110\n'
                                            '        },\n'
                                            '        "bw_min" : 53220,\n'
                                            '        "bw_max" : 53390,\n'
                                            '        "bw_agg" : 100.000000,\n'
                                            '        "bw_mean" : '
                                            '53313.285714,\n'
                                            '        "bw_dev" : 55.447034,\n'
                                            '        "bw_samples" : 14,\n'
                                            '        "iops_min" : 106440,\n'
                                            '        "iops_max" : 106780,\n'
                                            '        "iops_mean" : '
                                            '106626.714286,\n'
                                            '        "iops_stddev" : '
                                            '111.025589,\n'
                                            '        "iops_samples" : 14\n'
                                            '      },\n'
                                            '      "write" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "trim" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "sync" : {\n'
                                            '        "total_ios" : 0,\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        }\n'
                                            '      },\n'
                                            '      "job_runtime" : 7000,\n'
                                            '      "usr_cpu" : 13.171429,\n'
                                            '      "sys_cpu" : 34.214286,\n'
                                            '      "ctx" : 746110,\n'
                                            '      "majf" : 0,\n'
                                            '      "minf" : 0,\n'
                                            '      "iodepth_level" : {\n'
                                            '        "1" : 100.000000,\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_submit" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_complete" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ns" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.010000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_us" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.010000,\n'
                                            '        "10" : 99.015829,\n'
                                            '        "20" : 0.965541,\n'
                                            '        "50" : 0.010000,\n'
                                            '        "100" : 0.013403,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.010000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ms" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000,\n'
                                            '        "2000" : 0.000000,\n'
                                            '        ">=2000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_depth" : 1,\n'
                                            '      "latency_target" : 0,\n'
                                            '      "latency_percentile" : '
                                            '100.000000,\n'
                                            '      "latency_window" : 0\n'
                                            '    }\n'
                                            '  ]\n'
                                            '}\n'
                                            '\n'
                                            'libaio-xnvme: (groupid=0, '
                                            'jobs=1): err= 0: pid=6602: Wed '
                                            'Oct 26 15:06:53 2022\n'
                                            '  read: IOPS=107k, BW=52.0MiB/s '
                                            '(54.6MB/s)(364MiB/7001msec)\n'
                                            '    slat (nsec): min=1743, '
                                            'max=20578, avg=1853.32, '
                                            'stdev=84.94\n'
                                            '    clat (nsec): min=571, '
                                            'max=462377, avg=7237.21, '
                                            'stdev=1091.43\n'
                                            '     lat (usec): min=6, max=464, '
                                            'avg= 9.09, stdev= 1.10\n'
                                            '    clat percentiles (nsec):\n'
                                            '     |  1.00th=[ 6816],  5.00th=[ '
                                            '6944], 10.00th=[ 7008], 20.00th=[ '
                                            '7072],\n'
                                            '     | 30.00th=[ 7072], 40.00th=[ '
                                            '7072], 50.00th=[ 7136], 60.00th=[ '
                                            '7200],\n'
                                            '     | 70.00th=[ 7200], 80.00th=[ '
                                            '7264], 90.00th=[ 7328], 95.00th=[ '
                                            '7456],\n'
                                            '     | 99.00th=[ 9920], '
                                            '99.50th=[13632], 99.90th=[15424], '
                                            '99.95th=[16064],\n'
                                            '     | 99.99th=[57600]\n'
                                            '   bw (  KiB/s): min=53220, '
                                            'max=53390, per=100.00%, '
                                            'avg=53313.29, stdev=55.45, '
                                            'samples=14\n'
                                            '   iops        : min=106440, '
                                            'max=106780, avg=106626.71, '
                                            'stdev=111.03, samples=14\n'
                                            '  lat (nsec)   : 750=0.01%\n'
                                            '  lat (usec)   : 4=0.01%, '
                                            '10=99.02%, 20=0.97%, 50=0.01%, '
                                            '100=0.01%\n'
                                            '  lat (usec)   : 500=0.01%\n'
                                            '  cpu          : usr=13.17%, '
                                            'sys=34.21%, ctx=746110, majf=0, '
                                            'minf=0\n'
                                            '  IO depths    : 1=100.0%, '
                                            '2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, >=64=0.0%\n'
                                            '     submit    : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     complete  : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     issued rwts: '
                                            'total=746110,0,0,0 short=0,0,0,0 '
                                            'dropped=0,0,0,0\n'
                                            '     latency   : target=0, '
                                            'window=0, percentile=100.00%, '
                                            'depth=1\n'
                                            '\n'
                                            'Run status group 0 (all jobs):\n'
                                            '   READ: bw=52.0MiB/s (54.6MB/s), '
                                            '52.0MiB/s-52.0MiB/s '
                                            '(54.6MB/s-54.6MB/s), io=364MiB '
                                            '(382MB), run=7001-7001msec\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_99.output'),
                                  'state': {'begin': 1666782523.9498014,
                                            'cmd': 'cat '
                                                   '/tmp/fio-output_512_1_libaio-xnvme_2.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.8183553218841553,
                                            'end': 1666782524.7681568,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_99.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_99.state')}}},
 'id': '003_fio',
 'name': 'fio',
 'nr': 3,
 'status': {'elapsed': 1946.3082365989685,
            'failed': 0,
            'passed': 1,
            'skipped': 0,
            'started': 0.0},
 'uses': 'fioe',
 'with': {'iodepths': [1, 2, 4, 8], 'iosizes': ['512'], 'repetitions': 3}}
$ rm /tmp/fio-output_512_1_io_uring_cmd-reference_0.txt
rm: cannot remove '/tmp/fio-output_512_1_io_uring_cmd-reference_0.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring_cmd-reference" --rw="randread" --size="12G" --iodepth="1" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring_cmd" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_512_1_io_uring_cmd-reference_0.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_1_io_uring_cmd-reference_0.txt
io_uring_cmd-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=io_uring_cmd, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666789148,
  "timestamp_ms" : 1666789148400,
  "time" : "Wed Oct 26 14:59:08 2022",
  "jobs" : [
    {
      "jobname" : "io_uring_cmd-reference",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring_cmd-reference",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "1",
        "bs" : "512",
        "direct" : "0",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "io_uring_cmd",
        "filename" : "/dev/ng0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 394994176,
        "io_kbytes" : 385736,
        "bw_bytes" : 56419679,
        "bw" : 55097,
        "iops" : 110194.686473,
        "runtime" : 7001,
        "total_ios" : 771473,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1362,
          "max" : 5961,
          "mean" : 1460.212779,
          "stddev" : 51.214525,
          "N" : 771473
        },
        "clat_ns" : {
          "min" : 2955,
          "max" : 117891,
          "mean" : 7271.748888,
          "stddev" : 971.362791,
          "N" : 771473,
          "percentile" : {
            "1.000000" : 6752,
            "5.000000" : 7008,
            "10.000000" : 7008,
            "20.000000" : 7072,
            "30.000000" : 7072,
            "40.000000" : 7136,
            "50.000000" : 7200,
            "60.000000" : 7200,
            "70.000000" : 7264,
            "80.000000" : 7264,
            "90.000000" : 7392,
            "95.000000" : 7456,
            "99.000000" : 10432,
            "99.500000" : 13632,
            "99.900000" : 15680,
            "99.950000" : 15808,
            "99.990000" : 57600
          }
        },
        "lat_ns" : {
          "min" : 7354,
          "max" : 119324,
          "mean" : 8731.961667,
          "stddev" : 973.813190,
          "N" : 771473
        },
        "bw_min" : 54956,
        "bw_max" : 55331,
        "bw_agg" : 100.000000,
        "bw_mean" : 55107.071429,
        "bw_dev" : 106.053299,
        "bw_samples" : 14,
        "iops_min" : 109912,
        "iops_max" : 110662,
        "iops_mean" : 110214.142857,
        "iops_stddev" : 212.106598,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 8.028571,
      "sys_cpu" : 35.128571,
      "ctx" : 771473,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 100.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010000,
        "10" : 98.671243,
        "20" : 1.310610,
        "50" : 0.010000,
        "100" : 0.012703,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 1,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring_cmd-reference: (groupid=0, jobs=1): err= 0: pid=4935: Wed Oct 26 14:59:08 2022
  read: IOPS=110k, BW=53.8MiB/s (56.4MB/s)(377MiB/7001msec)
    slat (nsec): min=1362, max=5961, avg=1460.21, stdev=51.21
    clat (nsec): min=2955, max=117891, avg=7271.75, stdev=971.36
     lat (nsec): min=7354, max=119324, avg=8731.96, stdev=973.81
    clat percentiles (nsec):
     |  1.00th=[ 6752],  5.00th=[ 7008], 10.00th=[ 7008], 20.00th=[ 7072],
     | 30.00th=[ 7072], 40.00th=[ 7136], 50.00th=[ 7200], 60.00th=[ 7200],
     | 70.00th=[ 7264], 80.00th=[ 7264], 90.00th=[ 7392], 95.00th=[ 7456],
     | 99.00th=[10432], 99.50th=[13632], 99.90th=[15680], 99.95th=[15808],
     | 99.99th=[57600]
   bw (  KiB/s): min=54956, max=55331, per=100.00%, avg=55107.07, stdev=106.05, samples=14
   iops        : min=109912, max=110662, avg=110214.14, stdev=212.11, samples=14
  lat (usec)   : 4=0.01%, 10=98.67%, 20=1.31%, 50=0.01%, 100=0.01%
  lat (usec)   : 250=0.01%
  cpu          : usr=8.03%, sys=35.13%, ctx=771473, majf=0, minf=0
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=771473,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=53.8MiB/s (56.4MB/s), 53.8MiB/s-53.8MiB/s (56.4MB/s-56.4MB/s), io=377MiB (395MB), run=7001-7001msec

$ rm /tmp/fio-output_512_1_io_uring_cmd-reference_1.txt
rm: cannot remove '/tmp/fio-output_512_1_io_uring_cmd-reference_1.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring_cmd-reference" --rw="randread" --size="12G" --iodepth="1" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring_cmd" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_512_1_io_uring_cmd-reference_1.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_1_io_uring_cmd-reference_1.txt
io_uring_cmd-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=io_uring_cmd, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666789162,
  "timestamp_ms" : 1666789162269,
  "time" : "Wed Oct 26 14:59:22 2022",
  "jobs" : [
    {
      "jobname" : "io_uring_cmd-reference",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring_cmd-reference",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "1",
        "bs" : "512",
        "direct" : "0",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "io_uring_cmd",
        "filename" : "/dev/ng0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 394686464,
        "io_kbytes" : 385436,
        "bw_bytes" : 56383780,
        "bw" : 55062,
        "iops" : 110124.571429,
        "runtime" : 7000,
        "total_ios" : 770872,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1402,
          "max" : 13765,
          "mean" : 1493.597215,
          "stddev" : 57.341593,
          "N" : 770872
        },
        "clat_ns" : {
          "min" : 171,
          "max" : 77646,
          "mean" : 7265.220271,
          "stddev" : 980.277945,
          "N" : 770872,
          "percentile" : {
            "1.000000" : 6752,
            "5.000000" : 7008,
            "10.000000" : 7008,
            "20.000000" : 7072,
            "30.000000" : 7072,
            "40.000000" : 7136,
            "50.000000" : 7200,
            "60.000000" : 7200,
            "70.000000" : 7264,
            "80.000000" : 7264,
            "90.000000" : 7392,
            "95.000000" : 7456,
            "99.000000" : 10560,
            "99.500000" : 13504,
            "99.900000" : 15680,
            "99.950000" : 15808,
            "99.990000" : 59136
          }
        },
        "lat_ns" : {
          "min" : 7434,
          "max" : 79108,
          "mean" : 8758.817486,
          "stddev" : 983.505773,
          "N" : 770872
        },
        "bw_min" : 54999,
        "bw_max" : 55216,
        "bw_agg" : 100.000000,
        "bw_mean" : 55079.642857,
        "bw_dev" : 51.880939,
        "bw_samples" : 14,
        "iops_min" : 109998,
        "iops_max" : 110432,
        "iops_mean" : 110159.285714,
        "iops_stddev" : 103.761879,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 6.757143,
      "sys_cpu" : 36.657143,
      "ctx" : 770872,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 100.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010000,
        "10" : 98.658921,
        "20" : 1.323436,
        "50" : 0.010000,
        "100" : 0.013751,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 1,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring_cmd-reference: (groupid=0, jobs=1): err= 0: pid=4981: Wed Oct 26 14:59:22 2022
  read: IOPS=110k, BW=53.8MiB/s (56.4MB/s)(376MiB/7000msec)
    slat (nsec): min=1402, max=13765, avg=1493.60, stdev=57.34
    clat (nsec): min=171, max=77646, avg=7265.22, stdev=980.28
     lat (nsec): min=7434, max=79108, avg=8758.82, stdev=983.51
    clat percentiles (nsec):
     |  1.00th=[ 6752],  5.00th=[ 7008], 10.00th=[ 7008], 20.00th=[ 7072],
     | 30.00th=[ 7072], 40.00th=[ 7136], 50.00th=[ 7200], 60.00th=[ 7200],
     | 70.00th=[ 7264], 80.00th=[ 7264], 90.00th=[ 7392], 95.00th=[ 7456],
     | 99.00th=[10560], 99.50th=[13504], 99.90th=[15680], 99.95th=[15808],
     | 99.99th=[59136]
   bw (  KiB/s): min=54999, max=55216, per=100.00%, avg=55079.64, stdev=51.88, samples=14
   iops        : min=109998, max=110432, avg=110159.29, stdev=103.76, samples=14
  lat (nsec)   : 250=0.01%
  lat (usec)   : 4=0.01%, 10=98.66%, 20=1.32%, 50=0.01%, 100=0.01%
  cpu          : usr=6.76%, sys=36.66%, ctx=770872, majf=0, minf=0
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=770872,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=53.8MiB/s (56.4MB/s), 53.8MiB/s-53.8MiB/s (56.4MB/s-56.4MB/s), io=376MiB (395MB), run=7000-7000msec

$ rm /tmp/fio-output_512_1_io_uring_cmd-reference_2.txt
rm: cannot remove '/tmp/fio-output_512_1_io_uring_cmd-reference_2.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring_cmd-reference" --rw="randread" --size="12G" --iodepth="1" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring_cmd" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_512_1_io_uring_cmd-reference_2.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_1_io_uring_cmd-reference_2.txt
io_uring_cmd-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=io_uring_cmd, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666789176,
  "timestamp_ms" : 1666789176144,
  "time" : "Wed Oct 26 14:59:36 2022",
  "jobs" : [
    {
      "jobname" : "io_uring_cmd-reference",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring_cmd-reference",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "1",
        "bs" : "512",
        "direct" : "0",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "io_uring_cmd",
        "filename" : "/dev/ng0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 397914112,
        "io_kbytes" : 388588,
        "bw_bytes" : 56836753,
        "bw" : 55504,
        "iops" : 111009.141551,
        "runtime" : 7001,
        "total_ios" : 777175,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1352,
          "max" : 16331,
          "mean" : 1435.588716,
          "stddev" : 64.972788,
          "N" : 777176
        },
        "clat_ns" : {
          "min" : 200,
          "max" : 118251,
          "mean" : 7248.129245,
          "stddev" : 1059.290764,
          "N" : 777175,
          "percentile" : {
            "1.000000" : 6752,
            "5.000000" : 6944,
            "10.000000" : 7008,
            "20.000000" : 7008,
            "30.000000" : 7072,
            "40.000000" : 7072,
            "50.000000" : 7136,
            "60.000000" : 7200,
            "70.000000" : 7200,
            "80.000000" : 7264,
            "90.000000" : 7328,
            "95.000000" : 7456,
            "99.000000" : 10688,
            "99.500000" : 13632,
            "99.900000" : 15680,
            "99.950000" : 15936,
            "99.990000" : 60672
          }
        },
        "lat_ns" : {
          "min" : 7274,
          "max" : 119724,
          "mean" : 8683.718016,
          "stddev" : 1062.030108,
          "N" : 777175
        },
        "bw_min" : 55395,
        "bw_max" : 55748,
        "bw_agg" : 100.000000,
        "bw_mean" : 55535.571429,
        "bw_dev" : 121.563984,
        "bw_samples" : 14,
        "iops_min" : 110790,
        "iops_max" : 111496,
        "iops_mean" : 111071.285714,
        "iops_stddev" : 243.041443,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 7.557143,
      "sys_cpu" : 35.600000,
      "ctx" : 777179,
      "majf" : 0,
      "minf" : 1,
      "iodepth_level" : {
        "1" : 100.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.010000,
        "4" : 0.010000,
        "10" : 98.510245,
        "20" : 1.465307,
        "50" : 0.010000,
        "100" : 0.017757,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 1,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring_cmd-reference: (groupid=0, jobs=1): err= 0: pid=5027: Wed Oct 26 14:59:36 2022
  read: IOPS=111k, BW=54.2MiB/s (56.8MB/s)(379MiB/7001msec)
    slat (nsec): min=1352, max=16331, avg=1435.59, stdev=64.97
    clat (nsec): min=200, max=118251, avg=7248.13, stdev=1059.29
     lat (usec): min=7, max=119, avg= 8.68, stdev= 1.06
    clat percentiles (nsec):
     |  1.00th=[ 6752],  5.00th=[ 6944], 10.00th=[ 7008], 20.00th=[ 7008],
     | 30.00th=[ 7072], 40.00th=[ 7072], 50.00th=[ 7136], 60.00th=[ 7200],
     | 70.00th=[ 7200], 80.00th=[ 7264], 90.00th=[ 7328], 95.00th=[ 7456],
     | 99.00th=[10688], 99.50th=[13632], 99.90th=[15680], 99.95th=[15936],
     | 99.99th=[60672]
   bw (  KiB/s): min=55395, max=55748, per=100.00%, avg=55535.57, stdev=121.56, samples=14
   iops        : min=110790, max=111496, avg=111071.29, stdev=243.04, samples=14
  lat (nsec)   : 250=0.01%
  lat (usec)   : 2=0.01%, 4=0.01%, 10=98.51%, 20=1.47%, 50=0.01%
  lat (usec)   : 100=0.02%, 250=0.01%
  cpu          : usr=7.56%, sys=35.60%, ctx=777179, majf=0, minf=1
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=777175,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=54.2MiB/s (56.8MB/s), 54.2MiB/s-54.2MiB/s (56.8MB/s-56.8MB/s), io=379MiB (398MB), run=7001-7001msec

$ rm /tmp/fio-output_512_1_io_uring_cmd-xnvme_0.txt
rm: cannot remove '/tmp/fio-output_512_1_io_uring_cmd-xnvme_0.txt': No such file or directory

$ rm /tmp/fio-output_512_2_io_uring_cmd-reference_0.txt
rm: cannot remove '/tmp/fio-output_512_2_io_uring_cmd-reference_0.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring_cmd-reference" --rw="randread" --size="12G" --iodepth="2" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring_cmd" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_512_2_io_uring_cmd-reference_0.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_2_io_uring_cmd-reference_0.txt
io_uring_cmd-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=io_uring_cmd, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666789627,
  "timestamp_ms" : 1666789627760,
  "time" : "Wed Oct 26 15:07:07 2022",
  "jobs" : [
    {
      "jobname" : "io_uring_cmd-reference",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring_cmd-reference",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "2",
        "bs" : "512",
        "direct" : "0",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "io_uring_cmd",
        "filename" : "/dev/ng0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 762248704,
        "io_kbytes" : 744383,
        "bw_bytes" : 108877118,
        "bw" : 106325,
        "iops" : 212650.478503,
        "runtime" : 7001,
        "total_ios" : 1488766,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1172,
          "max" : 21300,
          "mean" : 1340.914016,
          "stddev" : 138.459284,
          "N" : 1488766
        },
        "clat_ns" : {
          "min" : 151,
          "max" : 118081,
          "mean" : 7781.440333,
          "stddev" : 2292.078431,
          "N" : 1488767,
          "percentile" : {
            "1.000000" : 5664,
            "5.000000" : 5792,
            "10.000000" : 5792,
            "20.000000" : 5856,
            "30.000000" : 5856,
            "40.000000" : 5920,
            "50.000000" : 9024,
            "60.000000" : 9152,
            "70.000000" : 9280,
            "80.000000" : 9280,
            "90.000000" : 9408,
            "95.000000" : 9536,
            "99.000000" : 16768,
            "99.500000" : 18048,
            "99.900000" : 19072,
            "99.950000" : 23936,
            "99.990000" : 64768
          }
        },
        "lat_ns" : {
          "min" : 5661,
          "max" : 119274,
          "mean" : 9122.354383,
          "stddev" : 2373.673269,
          "N" : 1488767
        },
        "bw_min" : 106162,
        "bw_max" : 106787,
        "bw_agg" : 100.000000,
        "bw_mean" : 106405.307692,
        "bw_dev" : 184.502206,
        "bw_samples" : 13,
        "iops_min" : 212324,
        "iops_max" : 213575,
        "iops_mean" : 212810.769231,
        "iops_stddev" : 369.150185,
        "iops_samples" : 13
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 14.371429,
      "sys_cpu" : 46.300000,
      "ctx" : 746929,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 100.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010000,
        "10" : 97.396434,
        "20" : 2.514230,
        "50" : 0.056221,
        "100" : 0.025928,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 2,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring_cmd-reference: (groupid=0, jobs=1): err= 0: pid=6649: Wed Oct 26 15:07:07 2022
  read: IOPS=213k, BW=104MiB/s (109MB/s)(727MiB/7001msec)
    slat (nsec): min=1172, max=21300, avg=1340.91, stdev=138.46
    clat (nsec): min=151, max=118081, avg=7781.44, stdev=2292.08
     lat (usec): min=5, max=119, avg= 9.12, stdev= 2.37
    clat percentiles (nsec):
     |  1.00th=[ 5664],  5.00th=[ 5792], 10.00th=[ 5792], 20.00th=[ 5856],
     | 30.00th=[ 5856], 40.00th=[ 5920], 50.00th=[ 9024], 60.00th=[ 9152],
     | 70.00th=[ 9280], 80.00th=[ 9280], 90.00th=[ 9408], 95.00th=[ 9536],
     | 99.00th=[16768], 99.50th=[18048], 99.90th=[19072], 99.95th=[23936],
     | 99.99th=[64768]
   bw (  KiB/s): min=106162, max=106787, per=100.00%, avg=106405.31, stdev=184.50, samples=13
   iops        : min=212324, max=213575, avg=212810.77, stdev=369.15, samples=13
  lat (nsec)   : 250=0.01%
  lat (usec)   : 4=0.01%, 10=97.40%, 20=2.51%, 50=0.06%, 100=0.03%
  lat (usec)   : 250=0.01%
  cpu          : usr=14.37%, sys=46.30%, ctx=746929, majf=0, minf=0
  IO depths    : 1=0.0%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1488766,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=2

Run status group 0 (all jobs):
   READ: bw=104MiB/s (109MB/s), 104MiB/s-104MiB/s (109MB/s-109MB/s), io=727MiB (762MB), run=7001-7001msec

$ rm /tmp/fio-output_512_2_io_uring_cmd-reference_1.txt
rm: cannot remove '/tmp/fio-output_512_2_io_uring_cmd-reference_1.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring_cmd-reference" --rw="randread" --size="12G" --iodepth="2" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring_cmd" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_512_2_io_uring_cmd-reference_1.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_2_io_uring_cmd-reference_1.txt
io_uring_cmd-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=io_uring_cmd, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666789641,
  "timestamp_ms" : 1666789641533,
  "time" : "Wed Oct 26 15:07:21 2022",
  "jobs" : [
    {
      "jobname" : "io_uring_cmd-reference",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring_cmd-reference",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "2",
        "bs" : "512",
        "direct" : "0",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "io_uring_cmd",
        "filename" : "/dev/ng0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 763893760,
        "io_kbytes" : 745990,
        "bw_bytes" : 109112092,
        "bw" : 106554,
        "iops" : 213109.270104,
        "runtime" : 7001,
        "total_ios" : 1491978,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1162,
          "max" : 46527,
          "mean" : 1340.729218,
          "stddev" : 143.970424,
          "N" : 1491979
        },
        "clat_ns" : {
          "min" : 150,
          "max" : 118172,
          "mean" : 7760.151302,
          "stddev" : 2264.115071,
          "N" : 1491979,
          "percentile" : {
            "1.000000" : 5600,
            "5.000000" : 5792,
            "10.000000" : 5792,
            "20.000000" : 5856,
            "30.000000" : 5856,
            "40.000000" : 5920,
            "50.000000" : 8896,
            "60.000000" : 9152,
            "70.000000" : 9280,
            "80.000000" : 9280,
            "90.000000" : 9408,
            "95.000000" : 9536,
            "99.000000" : 16768,
            "99.500000" : 18048,
            "99.900000" : 18816,
            "99.950000" : 23680,
            "99.990000" : 60672
          }
        },
        "lat_ns" : {
          "min" : 5590,
          "max" : 119384,
          "mean" : 9100.880520,
          "stddev" : 2350.258010,
          "N" : 1491979
        },
        "bw_min" : 106122,
        "bw_max" : 106923,
        "bw_agg" : 100.000000,
        "bw_mean" : 106573.500000,
        "bw_dev" : 240.951559,
        "bw_samples" : 14,
        "iops_min" : 212244,
        "iops_max" : 213846,
        "iops_mean" : 213147.000000,
        "iops_stddev" : 481.903118,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 14.057143,
      "sys_cpu" : 46.657143,
      "ctx" : 748714,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 100.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010000,
        "10" : 97.423286,
        "20" : 2.503053,
        "50" : 0.046985,
        "100" : 0.020979,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 2,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring_cmd-reference: (groupid=0, jobs=1): err= 0: pid=6695: Wed Oct 26 15:07:21 2022
  read: IOPS=213k, BW=104MiB/s (109MB/s)(729MiB/7001msec)
    slat (nsec): min=1162, max=46527, avg=1340.73, stdev=143.97
    clat (nsec): min=150, max=118172, avg=7760.15, stdev=2264.12
     lat (usec): min=5, max=119, avg= 9.10, stdev= 2.35
    clat percentiles (nsec):
     |  1.00th=[ 5600],  5.00th=[ 5792], 10.00th=[ 5792], 20.00th=[ 5856],
     | 30.00th=[ 5856], 40.00th=[ 5920], 50.00th=[ 8896], 60.00th=[ 9152],
     | 70.00th=[ 9280], 80.00th=[ 9280], 90.00th=[ 9408], 95.00th=[ 9536],
     | 99.00th=[16768], 99.50th=[18048], 99.90th=[18816], 99.95th=[23680],
     | 99.99th=[60672]
   bw (  KiB/s): min=106122, max=106923, per=100.00%, avg=106573.50, stdev=240.95, samples=14
   iops        : min=212244, max=213846, avg=213147.00, stdev=481.90, samples=14
  lat (nsec)   : 250=0.01%
  lat (usec)   : 4=0.01%, 10=97.42%, 20=2.50%, 50=0.05%, 100=0.02%
  lat (usec)   : 250=0.01%
  cpu          : usr=14.06%, sys=46.66%, ctx=748714, majf=0, minf=0
  IO depths    : 1=0.0%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1491978,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=2

Run status group 0 (all jobs):
   READ: bw=104MiB/s (109MB/s), 104MiB/s-104MiB/s (109MB/s-109MB/s), io=729MiB (764MB), run=7001-7001msec

$ rm /tmp/fio-output_512_2_io_uring_cmd-reference_2.txt
rm: cannot remove '/tmp/fio-output_512_2_io_uring_cmd-reference_2.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring_cmd-reference" --rw="randread" --size="12G" --iodepth="2" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring_cmd" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_512_2_io_uring_cmd-reference_2.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_2_io_uring_cmd-reference_2.txt
io_uring_cmd-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=io_uring_cmd, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666789655,
  "timestamp_ms" : 1666789655332,
  "time" : "Wed Oct 26 15:07:35 2022",
  "jobs" : [
    {
      "jobname" : "io_uring_cmd-reference",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring_cmd-reference",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "2",
        "bs" : "512",
        "direct" : "0",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "io_uring_cmd",
        "filename" : "/dev/ng0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 764250112,
        "io_kbytes" : 746338,
        "bw_bytes" : 109162992,
        "bw" : 106604,
        "iops" : 213208.827310,
        "runtime" : 7001,
        "total_ios" : 1492675,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1162,
          "max" : 25678,
          "mean" : 1346.666306,
          "stddev" : 139.012440,
          "N" : 1492675
        },
        "clat_ns" : {
          "min" : 90,
          "max" : 244338,
          "mean" : 7752.355304,
          "stddev" : 2295.470051,
          "N" : 1492676,
          "percentile" : {
            "1.000000" : 5600,
            "5.000000" : 5728,
            "10.000000" : 5792,
            "20.000000" : 5792,
            "30.000000" : 5856,
            "40.000000" : 5920,
            "50.000000" : 9024,
            "60.000000" : 9152,
            "70.000000" : 9280,
            "80.000000" : 9280,
            "90.000000" : 9408,
            "95.000000" : 9536,
            "99.000000" : 16768,
            "99.500000" : 18048,
            "99.900000" : 18816,
            "99.950000" : 23680,
            "99.990000" : 63232
          }
        },
        "lat_ns" : {
          "min" : 5621,
          "max" : 245951,
          "mean" : 9099.021662,
          "stddev" : 2381.626970,
          "N" : 1492676
        },
        "bw_min" : 106186,
        "bw_max" : 107335,
        "bw_agg" : 100.000000,
        "bw_mean" : 106623.785714,
        "bw_dev" : 356.455883,
        "bw_samples" : 14,
        "iops_min" : 212372,
        "iops_max" : 214671,
        "iops_mean" : 213247.642857,
        "iops_stddev" : 713.065280,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 12.200000,
      "sys_cpu" : 48.442857,
      "ctx" : 748942,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 100.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.010000,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.010000,
        "4" : 0.010000,
        "10" : 97.413905,
        "20" : 2.505435,
        "50" : 0.049776,
        "100" : 0.022912,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 2,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring_cmd-reference: (groupid=0, jobs=1): err= 0: pid=6741: Wed Oct 26 15:07:35 2022
  read: IOPS=213k, BW=104MiB/s (109MB/s)(729MiB/7001msec)
    slat (nsec): min=1162, max=25678, avg=1346.67, stdev=139.01
    clat (nsec): min=90, max=244338, avg=7752.36, stdev=2295.47
     lat (usec): min=5, max=245, avg= 9.10, stdev= 2.38
    clat percentiles (nsec):
     |  1.00th=[ 5600],  5.00th=[ 5728], 10.00th=[ 5792], 20.00th=[ 5792],
     | 30.00th=[ 5856], 40.00th=[ 5920], 50.00th=[ 9024], 60.00th=[ 9152],
     | 70.00th=[ 9280], 80.00th=[ 9280], 90.00th=[ 9408], 95.00th=[ 9536],
     | 99.00th=[16768], 99.50th=[18048], 99.90th=[18816], 99.95th=[23680],
     | 99.99th=[63232]
   bw (  KiB/s): min=106186, max=107335, per=100.00%, avg=106623.79, stdev=356.46, samples=14
   iops        : min=212372, max=214671, avg=213247.64, stdev=713.07, samples=14
  lat (nsec)   : 100=0.01%, 250=0.01%
  lat (usec)   : 2=0.01%, 4=0.01%, 10=97.41%, 20=2.51%, 50=0.05%
  lat (usec)   : 100=0.02%, 250=0.01%
  cpu          : usr=12.20%, sys=48.44%, ctx=748942, majf=0, minf=0
  IO depths    : 1=0.0%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1492675,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=2

Run status group 0 (all jobs):
   READ: bw=104MiB/s (109MB/s), 104MiB/s-104MiB/s (109MB/s-109MB/s), io=729MiB (764MB), run=7001-7001msec

$ rm /tmp/fio-output_512_2_io_uring_cmd-xnvme_0.txt
rm: cannot remove '/tmp/fio-output_512_2_io_uring_cmd-xnvme_0.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring_cmd-xnvme" --rw="randread" --size="12G" --iodepth="1" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring_cmd" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_512_1_io_uring_cmd-xnvme_0.txt" --cpus_allowed="1"
$ /opt/fio/bin/fio --name="io_uring_cmd-xnvme" --rw="randread" --size="12G" --iodepth="2" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring_cmd" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_512_2_io_uring_cmd-xnvme_0.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_2_io_uring_cmd-xnvme_0.txt
io_uring_cmd-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666789669,
  "timestamp_ms" : 1666789669194,
  "time" : "Wed Oct 26 15:07:49 2022",
  "jobs" : [
    {
      "jobname" : "io_uring_cmd-xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring_cmd-xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "2",
        "bs" : "512",
        "direct" : "0",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
        "xnvme_be" : "linux",
        "xnvme_async" : "io_uring_cmd",
        "xnvme_sync" : "nvme",
        "xnvme_admin" : "nvme",
        "xnvme_dev_nsid" : "1",
        "filename" : "/dev/ng0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 760012288,
        "io_kbytes" : 742199,
        "bw_bytes" : 108557675,
        "bw" : 106013,
        "iops" : 212026.424796,
        "runtime" : 7001,
        "total_ios" : 1484397,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1222,
          "max" : 22522,
          "mean" : 1423.243689,
          "stddev" : 173.920884,
          "N" : 1484398
        },
        "clat_ns" : {
          "min" : 151,
          "max" : 115626,
          "mean" : 7748.459844,
          "stddev" : 2130.869047,
          "N" : 1484398,
          "percentile" : {
            "1.000000" : 5536,
            "5.000000" : 5664,
            "10.000000" : 5728,
            "20.000000" : 5792,
            "30.000000" : 6176,
            "40.000000" : 7328,
            "50.000000" : 7520,
            "60.000000" : 7712,
            "70.000000" : 9280,
            "80.000000" : 9408,
            "90.000000" : 9536,
            "95.000000" : 9664,
            "99.000000" : 16768,
            "99.500000" : 18304,
            "99.900000" : 23936,
            "99.950000" : 24448,
            "99.990000" : 61184
          }
        },
        "lat_ns" : {
          "min" : 5751,
          "max" : 116909,
          "mean" : 9171.703695,
          "stddev" : 2188.738867,
          "N" : 1484398
        },
        "bw_min" : 105599,
        "bw_max" : 106438,
        "bw_agg" : 100.000000,
        "bw_mean" : 106032.142857,
        "bw_dev" : 187.439367,
        "bw_samples" : 14,
        "iops_min" : 211198,
        "iops_max" : 212876,
        "iops_mean" : 212064.285714,
        "iops_stddev" : 374.878735,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 15.985714,
      "sys_cpu" : 46.657143,
      "ctx" : 745370,
      "majf" : 0,
      "minf" : 1,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 100.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010000,
        "10" : 97.303619,
        "20" : 2.542110,
        "50" : 0.122811,
        "100" : 0.021288,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 2,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring_cmd-xnvme: (groupid=0, jobs=1): err= 0: pid=6787: Wed Oct 26 15:07:49 2022
  read: IOPS=212k, BW=104MiB/s (109MB/s)(725MiB/7001msec)
    slat (nsec): min=1222, max=22522, avg=1423.24, stdev=173.92
    clat (nsec): min=151, max=115626, avg=7748.46, stdev=2130.87
     lat (usec): min=5, max=116, avg= 9.17, stdev= 2.19
    clat percentiles (nsec):
     |  1.00th=[ 5536],  5.00th=[ 5664], 10.00th=[ 5728], 20.00th=[ 5792],
     | 30.00th=[ 6176], 40.00th=[ 7328], 50.00th=[ 7520], 60.00th=[ 7712],
     | 70.00th=[ 9280], 80.00th=[ 9408], 90.00th=[ 9536], 95.00th=[ 9664],
     | 99.00th=[16768], 99.50th=[18304], 99.90th=[23936], 99.95th=[24448],
     | 99.99th=[61184]
   bw (  KiB/s): min=105599, max=106438, per=100.00%, avg=106032.14, stdev=187.44, samples=14
   iops        : min=211198, max=212876, avg=212064.29, stdev=374.88, samples=14
  lat (nsec)   : 250=0.01%
  lat (usec)   : 4=0.01%, 10=97.30%, 20=2.54%, 50=0.12%, 100=0.02%
  lat (usec)   : 250=0.01%
  cpu          : usr=15.99%, sys=46.66%, ctx=745370, majf=0, minf=1
  IO depths    : 1=0.0%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1484397,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=2

Run status group 0 (all jobs):
   READ: bw=104MiB/s (109MB/s), 104MiB/s-104MiB/s (109MB/s-109MB/s), io=725MiB (760MB), run=7001-7001msec

$ rm /tmp/fio-output_512_2_io_uring_cmd-xnvme_1.txt
rm: cannot remove '/tmp/fio-output_512_2_io_uring_cmd-xnvme_1.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring_cmd-xnvme" --rw="randread" --size="12G" --iodepth="2" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring_cmd" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_512_2_io_uring_cmd-xnvme_1.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_2_io_uring_cmd-xnvme_1.txt
io_uring_cmd-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666789683,
  "timestamp_ms" : 1666789683112,
  "time" : "Wed Oct 26 15:08:03 2022",
  "jobs" : [
    {
      "jobname" : "io_uring_cmd-xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring_cmd-xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "2",
        "bs" : "512",
        "direct" : "0",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
        "xnvme_be" : "linux",
        "xnvme_async" : "io_uring_cmd",
        "xnvme_sync" : "nvme",
        "xnvme_admin" : "nvme",
        "xnvme_dev_nsid" : "1",
        "filename" : "/dev/ng0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 762302976,
        "io_kbytes" : 744436,
        "bw_bytes" : 108884870,
        "bw" : 106332,
        "iops" : 212665.619197,
        "runtime" : 7001,
        "total_ios" : 1488872,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1222,
          "max" : 13836,
          "mean" : 1386.090252,
          "stddev" : 141.744320,
          "N" : 1488872
        },
        "clat_ns" : {
          "min" : 1954,
          "max" : 79479,
          "mean" : 7760.864534,
          "stddev" : 2225.390339,
          "N" : 1488873,
          "percentile" : {
            "1.000000" : 5536,
            "5.000000" : 5728,
            "10.000000" : 5728,
            "20.000000" : 5792,
            "30.000000" : 5856,
            "40.000000" : 6496,
            "50.000000" : 7520,
            "60.000000" : 9152,
            "70.000000" : 9280,
            "80.000000" : 9408,
            "90.000000" : 9408,
            "95.000000" : 9536,
            "99.000000" : 16768,
            "99.500000" : 18304,
            "99.900000" : 22400,
            "99.950000" : 24192,
            "99.990000" : 61696
          }
        },
        "lat_ns" : {
          "min" : 5721,
          "max" : 80751,
          "mean" : 9146.954825,
          "stddev" : 2289.506518,
          "N" : 1488873
        },
        "bw_min" : 106032,
        "bw_max" : 106788,
        "bw_agg" : 100.000000,
        "bw_mean" : 106351.571429,
        "bw_dev" : 183.468176,
        "bw_samples" : 14,
        "iops_min" : 212064,
        "iops_max" : 213577,
        "iops_mean" : 212703.214286,
        "iops_stddev" : 367.119385,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 15.785714,
      "sys_cpu" : 46.285714,
      "ctx" : 747296,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 100.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.010000,
        "4" : 0.010000,
        "10" : 97.373313,
        "20" : 2.508879,
        "50" : 0.086441,
        "100" : 0.022433,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 2,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring_cmd-xnvme: (groupid=0, jobs=1): err= 0: pid=6834: Wed Oct 26 15:08:03 2022
  read: IOPS=213k, BW=104MiB/s (109MB/s)(727MiB/7001msec)
    slat (nsec): min=1222, max=13836, avg=1386.09, stdev=141.74
    clat (nsec): min=1954, max=79479, avg=7760.86, stdev=2225.39
     lat (nsec): min=5721, max=80751, avg=9146.95, stdev=2289.51
    clat percentiles (nsec):
     |  1.00th=[ 5536],  5.00th=[ 5728], 10.00th=[ 5728], 20.00th=[ 5792],
     | 30.00th=[ 5856], 40.00th=[ 6496], 50.00th=[ 7520], 60.00th=[ 9152],
     | 70.00th=[ 9280], 80.00th=[ 9408], 90.00th=[ 9408], 95.00th=[ 9536],
     | 99.00th=[16768], 99.50th=[18304], 99.90th=[22400], 99.95th=[24192],
     | 99.99th=[61696]
   bw (  KiB/s): min=106032, max=106788, per=100.00%, avg=106351.57, stdev=183.47, samples=14
   iops        : min=212064, max=213577, avg=212703.21, stdev=367.12, samples=14
  lat (usec)   : 2=0.01%, 4=0.01%, 10=97.37%, 20=2.51%, 50=0.09%
  lat (usec)   : 100=0.02%
  cpu          : usr=15.79%, sys=46.29%, ctx=747296, majf=0, minf=0
  IO depths    : 1=0.0%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1488872,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=2

Run status group 0 (all jobs):
   READ: bw=104MiB/s (109MB/s), 104MiB/s-104MiB/s (109MB/s-109MB/s), io=727MiB (762MB), run=7001-7001msec

$ rm /tmp/fio-output_512_2_io_uring_cmd-xnvme_2.txt
rm: cannot remove '/tmp/fio-output_512_2_io_uring_cmd-xnvme_2.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring_cmd-xnvme" --rw="randread" --size="12G" --iodepth="2" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring_cmd" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_512_2_io_uring_cmd-xnvme_2.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_2_io_uring_cmd-xnvme_2.txt
io_uring_cmd-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666789696,
  "timestamp_ms" : 1666789696879,
  "time" : "Wed Oct 26 15:08:16 2022",
  "jobs" : [
    {
      "jobname" : "io_uring_cmd-xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring_cmd-xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "2",
        "bs" : "512",
        "direct" : "0",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
        "xnvme_be" : "linux",
        "xnvme_async" : "io_uring_cmd",
        "xnvme_sync" : "nvme",
        "xnvme_admin" : "nvme",
        "xnvme_dev_nsid" : "1",
        "filename" : "/dev/ng0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 762843136,
        "io_kbytes" : 744964,
        "bw_bytes" : 108977590,
        "bw" : 106423,
        "iops" : 212846.714286,
        "runtime" : 7000,
        "total_ios" : 1489927,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1212,
          "max" : 34444,
          "mean" : 1402.238872,
          "stddev" : 161.624220,
          "N" : 1489927
        },
        "clat_ns" : {
          "min" : 180,
          "max" : 110517,
          "mean" : 7730.380643,
          "stddev" : 2149.553208,
          "N" : 1489928,
          "percentile" : {
            "1.000000" : 5536,
            "5.000000" : 5664,
            "10.000000" : 5728,
            "20.000000" : 5792,
            "30.000000" : 5920,
            "40.000000" : 7264,
            "50.000000" : 7520,
            "60.000000" : 7712,
            "70.000000" : 9280,
            "80.000000" : 9408,
            "90.000000" : 9408,
            "95.000000" : 9536,
            "99.000000" : 16768,
            "99.500000" : 18304,
            "99.900000" : 23424,
            "99.950000" : 24192,
            "99.990000" : 63744
          }
        },
        "lat_ns" : {
          "min" : 5690,
          "max" : 111760,
          "mean" : 9132.619582,
          "stddev" : 2205.875773,
          "N" : 1489928
        },
        "bw_min" : 106063,
        "bw_max" : 106683,
        "bw_agg" : 100.000000,
        "bw_mean" : 106427.071429,
        "bw_dev" : 168.812807,
        "bw_samples" : 14,
        "iops_min" : 212126,
        "iops_max" : 213366,
        "iops_mean" : 212854.142857,
        "iops_stddev" : 337.625614,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 14.314286,
      "sys_cpu" : 48.085714,
      "ctx" : 747909,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 100.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010000,
        "10" : 97.368864,
        "20" : 2.503680,
        "50" : 0.096649,
        "100" : 0.021276,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 2,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring_cmd-xnvme: (groupid=0, jobs=1): err= 0: pid=6880: Wed Oct 26 15:08:16 2022
  read: IOPS=213k, BW=104MiB/s (109MB/s)(728MiB/7000msec)
    slat (nsec): min=1212, max=34444, avg=1402.24, stdev=161.62
    clat (nsec): min=180, max=110517, avg=7730.38, stdev=2149.55
     lat (usec): min=5, max=111, avg= 9.13, stdev= 2.21
    clat percentiles (nsec):
     |  1.00th=[ 5536],  5.00th=[ 5664], 10.00th=[ 5728], 20.00th=[ 5792],
     | 30.00th=[ 5920], 40.00th=[ 7264], 50.00th=[ 7520], 60.00th=[ 7712],
     | 70.00th=[ 9280], 80.00th=[ 9408], 90.00th=[ 9408], 95.00th=[ 9536],
     | 99.00th=[16768], 99.50th=[18304], 99.90th=[23424], 99.95th=[24192],
     | 99.99th=[63744]
   bw (  KiB/s): min=106063, max=106683, per=100.00%, avg=106427.07, stdev=168.81, samples=14
   iops        : min=212126, max=213366, avg=212854.14, stdev=337.63, samples=14
  lat (nsec)   : 250=0.01%
  lat (usec)   : 4=0.01%, 10=97.37%, 20=2.50%, 50=0.10%, 100=0.02%
  lat (usec)   : 250=0.01%
  cpu          : usr=14.31%, sys=48.09%, ctx=747909, majf=0, minf=0
  IO depths    : 1=0.0%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1489927,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=2

Run status group 0 (all jobs):
   READ: bw=104MiB/s (109MB/s), 104MiB/s-104MiB/s (109MB/s-109MB/s), io=728MiB (763MB), run=7000-7000msec

$ rm /tmp/fio-output_512_2_io_uring_cmd-bdev_xnvme_0.txt
rm: cannot remove '/tmp/fio-output_512_2_io_uring_cmd-bdev_xnvme_0.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring_cmd-bdev_xnvme" --rw="randread" --size="12G" --iodepth="2" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_2_io_uring_cmd-bdev_xnvme_0.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_512_1_io_uring_cmd-xnvme_0.txt
io_uring_cmd-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666789189,
  "timestamp_ms" : 1666789189890,
  "time" : "Wed Oct 26 14:59:49 2022",
  "jobs" : [
    {
      "jobname" : "io_uring_cmd-xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring_cmd-xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "1",
        "bs" : "512",
        "direct" : "0",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
        "xnvme_be" : "linux",
        "xnvme_async" : "io_uring_cmd",
        "xnvme_sync" : "nvme",
        "xnvme_admin" : "nvme",
        "xnvme_dev_nsid" : "1",
        "filename" : "/dev/ng0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 394787328,
        "io_kbytes" : 385534,
        "bw_bytes" : 56390133,
        "bw" : 55068,
        "iops" : 110136.980431,
        "runtime" : 7001,
        "total_ios" : 771069,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1402,
          "max" : 10500,
          "mean" : 1496.382610,
          "stddev" : 66.312543,
          "N" : 771069
        },
        "clat_ns" : {
          "min" : 211,
          "max" : 101751,
          "mean" : 7283.762805,
          "stddev" : 1020.270135,
          "N" : 771069,
          "percentile" : {
            "1.000000" : 6752,
            "5.000000" : 7008,
            "10.000000" : 7072,
            "20.000000" : 7072,
            "30.000000" : 7072,
            "40.000000" : 7136,
            "50.000000" : 7200,
            "60.000000" : 7200,
            "70.000000" : 7264,
            "80.000000" : 7264,
            "90.000000" : 7392,
            "95.000000" : 7456,
            "99.000000" : 10560,
            "99.500000" : 13504,
            "99.900000" : 15680,
            "99.950000" : 15936,
            "99.990000" : 58624
          }
        },
        "lat_ns" : {
          "min" : 7324,
          "max" : 103243,
          "mean" : 8780.145415,
          "stddev" : 1023.065350,
          "N" : 771069
        },
        "bw_min" : 54847,
        "bw_max" : 55341,
        "bw_agg" : 100.000000,
        "bw_mean" : 55095.571429,
        "bw_dev" : 155.343646,
        "bw_samples" : 14,
        "iops_min" : 109694,
        "iops_max" : 110683,
        "iops_mean" : 110191.142857,
        "iops_stddev" : 310.701404,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 10.771429,
      "sys_cpu" : 33.328571,
      "ctx" : 771067,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 100.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.010000,
        "4" : 0.010000,
        "10" : 98.642275,
        "20" : 1.335289,
        "50" : 0.010000,
        "100" : 0.016211,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 1,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring_cmd-xnvme: (groupid=0, jobs=1): err= 0: pid=5073: Wed Oct 26 14:59:49 2022
  read: IOPS=110k, BW=53.8MiB/s (56.4MB/s)(376MiB/7001msec)
    slat (nsec): min=1402, max=10500, avg=1496.38, stdev=66.31
    clat (nsec): min=211, max=101751, avg=7283.76, stdev=1020.27
     lat (usec): min=7, max=103, avg= 8.78, stdev= 1.02
    clat percentiles (nsec):
     |  1.00th=[ 6752],  5.00th=[ 7008], 10.00th=[ 7072], 20.00th=[ 7072],
     | 30.00th=[ 7072], 40.00th=[ 7136], 50.00th=[ 7200], 60.00th=[ 7200],
     | 70.00th=[ 7264], 80.00th=[ 7264], 90.00th=[ 7392], 95.00th=[ 7456],
     | 99.00th=[10560], 99.50th=[13504], 99.90th=[15680], 99.95th=[15936],
     | 99.99th=[58624]
   bw (  KiB/s): min=54847, max=55341, per=100.00%, avg=55095.57, stdev=155.34, samples=14
   iops        : min=109694, max=110683, avg=110191.14, stdev=310.70, samples=14
  lat (nsec)   : 250=0.01%
  lat (usec)   : 2=0.01%, 4=0.01%, 10=98.64%, 20=1.34%, 50=0.01%
  lat (usec)   : 100=0.02%, 250=0.01%
  cpu          : usr=10.77%, sys=33.33%, ctx=771067, majf=0, minf=0
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=771069,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=53.8MiB/s (56.4MB/s), 53.8MiB/s-53.8MiB/s (56.4MB/s-56.4MB/s), io=376MiB (395MB), run=7001-7001msec

$ cat /tmp/fio-output_512_2_io_uring_cmd-bdev_xnvme_0.txt
io_uring_cmd-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666789712,
  "timestamp_ms" : 1666789712219,
  "time" : "Wed Oct 26 15:08:32 2022",
  "jobs" : [
    {
      "jobname" : "io_uring_cmd-bdev_xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring_cmd-bdev_xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "2",
        "bs" : "512",
        "direct" : "0",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",
        "filename" : "Nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 909902848,
        "io_kbytes" : 888577,
        "bw_bytes" : 129967554,
        "bw" : 126921,
        "iops" : 253842.736752,
        "runtime" : 7001,
        "total_ios" : 1777153,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 220,
          "max" : 16330,
          "mean" : 271.297074,
          "stddev" : 24.250548,
          "N" : 1777153
        },
        "clat_ns" : {
          "min" : 281,
          "max" : 121247,
          "mean" : 7381.103358,
          "stddev" : 1757.802881,
          "N" : 1777154,
          "percentile" : {
            "1.000000" : 5984,
            "5.000000" : 6240,
            "10.000000" : 6368,
            "20.000000" : 6624,
            "30.000000" : 7072,
            "40.000000" : 7136,
            "50.000000" : 7200,
            "60.000000" : 7264,
            "70.000000" : 7328,
            "80.000000" : 7520,
            "90.000000" : 8032,
            "95.000000" : 8256,
            "99.000000" : 16064,
            "99.500000" : 20352,
            "99.900000" : 22144,
            "99.950000" : 22400,
            "99.990000" : 63232
          }
        },
        "lat_ns" : {
          "min" : 5480,
          "max" : 121528,
          "mean" : 7652.400431,
          "stddev" : 1758.138025,
          "N" : 1777154
        },
        "bw_min" : 125453,
        "bw_max" : 129526,
        "bw_agg" : 100.000000,
        "bw_mean" : 126965.500000,
        "bw_dev" : 1491.929507,
        "bw_samples" : 14,
        "iops_min" : 250906,
        "iops_max" : 259052,
        "iops_mean" : 253931.000000,
        "iops_stddev" : 2984.065734,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 99.985714,
      "sys_cpu" : 0.000000,
      "ctx" : 17,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.100000,
        "2" : 99.985314,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.010000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010000,
        "10" : 97.055459,
        "20" : 2.406096,
        "50" : 0.517007,
        "100" : 0.020651,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 2,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring_cmd-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=6940: Wed Oct 26 15:08:32 2022
  read: IOPS=254k, BW=124MiB/s (130MB/s)(868MiB/7001msec)
    slat (nsec): min=220, max=16330, avg=271.30, stdev=24.25
    clat (nsec): min=281, max=121247, avg=7381.10, stdev=1757.80
     lat (usec): min=5, max=121, avg= 7.65, stdev= 1.76
    clat percentiles (nsec):
     |  1.00th=[ 5984],  5.00th=[ 6240], 10.00th=[ 6368], 20.00th=[ 6624],
     | 30.00th=[ 7072], 40.00th=[ 7136], 50.00th=[ 7200], 60.00th=[ 7264],
     | 70.00th=[ 7328], 80.00th=[ 7520], 90.00th=[ 8032], 95.00th=[ 8256],
     | 99.00th=[16064], 99.50th=[20352], 99.90th=[22144], 99.95th=[22400],
     | 99.99th=[63232]
   bw (  KiB/s): min=125453, max=129526, per=100.00%, avg=126965.50, stdev=1491.93, samples=14
   iops        : min=250906, max=259052, avg=253931.00, stdev=2984.07, samples=14
  lat (nsec)   : 500=0.01%
  lat (usec)   : 4=0.01%, 10=97.06%, 20=2.41%, 50=0.52%, 100=0.02%
  lat (usec)   : 250=0.01%
  cpu          : usr=99.99%, sys=0.00%, ctx=17, majf=0, minf=0
  IO depths    : 1=0.1%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1777153,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=2

Run status group 0 (all jobs):
   READ: bw=124MiB/s (130MB/s), 124MiB/s-124MiB/s (130MB/s-130MB/s), io=868MiB (910MB), run=7001-7001msec

$ rm /tmp/fio-output_512_2_io_uring_cmd-bdev_xnvme_1.txt
rm: cannot remove '/tmp/fio-output_512_2_io_uring_cmd-bdev_xnvme_1.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring_cmd-bdev_xnvme" --rw="randread" --size="12G" --iodepth="2" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_2_io_uring_cmd-bdev_xnvme_1.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_512_2_io_uring_cmd-bdev_xnvme_1.txt
io_uring_cmd-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666789727,
  "timestamp_ms" : 1666789727362,
  "time" : "Wed Oct 26 15:08:47 2022",
  "jobs" : [
    {
      "jobname" : "io_uring_cmd-bdev_xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring_cmd-bdev_xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "2",
        "bs" : "512",
        "direct" : "0",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",
        "filename" : "Nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 699008512,
        "io_kbytes" : 682625,
        "bw_bytes" : 99844095,
        "bw" : 97503,
        "iops" : 195007.856021,
        "runtime" : 7001,
        "total_ios" : 1365250,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 711,
          "max" : 12062,
          "mean" : 936.282706,
          "stddev" : 69.667152,
          "N" : 1365250
        },
        "clat_ns" : {
          "min" : 5841,
          "max" : 82906,
          "mean" : 9006.613612,
          "stddev" : 1552.777792,
          "N" : 1365251,
          "percentile" : {
            "1.000000" : 7264,
            "5.000000" : 8032,
            "10.000000" : 8512,
            "20.000000" : 8640,
            "30.000000" : 8768,
            "40.000000" : 8768,
            "50.000000" : 8768,
            "60.000000" : 8896,
            "70.000000" : 9024,
            "80.000000" : 9024,
            "90.000000" : 9280,
            "95.000000" : 9536,
            "99.000000" : 17024,
            "99.500000" : 18304,
            "99.900000" : 24704,
            "99.950000" : 25728,
            "99.990000" : 62208
          }
        },
        "lat_ns" : {
          "min" : 7314,
          "max" : 83837,
          "mean" : 9942.896300,
          "stddev" : 1554.645663,
          "N" : 1365251
        },
        "bw_min" : 97305,
        "bw_max" : 98105,
        "bw_agg" : 100.000000,
        "bw_mean" : 97526.428571,
        "bw_dev" : 203.481889,
        "bw_samples" : 14,
        "iops_min" : 194610,
        "iops_max" : 196210,
        "iops_mean" : 195052.857143,
        "iops_stddev" : 406.963778,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 99.985714,
      "sys_cpu" : 0.000000,
      "ctx" : 16,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.534627,
        "2" : 99.465373,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 96.686321,
        "20" : 2.956968,
        "50" : 0.336129,
        "100" : 0.020656,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 2,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring_cmd-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=6998: Wed Oct 26 15:08:47 2022
  read: IOPS=195k, BW=95.2MiB/s (99.8MB/s)(667MiB/7001msec)
    slat (nsec): min=711, max=12062, avg=936.28, stdev=69.67
    clat (nsec): min=5841, max=82906, avg=9006.61, stdev=1552.78
     lat (nsec): min=7314, max=83837, avg=9942.90, stdev=1554.65
    clat percentiles (nsec):
     |  1.00th=[ 7264],  5.00th=[ 8032], 10.00th=[ 8512], 20.00th=[ 8640],
     | 30.00th=[ 8768], 40.00th=[ 8768], 50.00th=[ 8768], 60.00th=[ 8896],
     | 70.00th=[ 9024], 80.00th=[ 9024], 90.00th=[ 9280], 95.00th=[ 9536],
     | 99.00th=[17024], 99.50th=[18304], 99.90th=[24704], 99.95th=[25728],
     | 99.99th=[62208]
   bw (  KiB/s): min=97305, max=98105, per=100.00%, avg=97526.43, stdev=203.48, samples=14
   iops        : min=194610, max=196210, avg=195052.86, stdev=406.96, samples=14
  lat (usec)   : 10=96.69%, 20=2.96%, 50=0.34%, 100=0.02%
  cpu          : usr=99.99%, sys=0.00%, ctx=16, majf=0, minf=0
  IO depths    : 1=0.5%, 2=99.5%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1365250,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=2

Run status group 0 (all jobs):
   READ: bw=95.2MiB/s (99.8MB/s), 95.2MiB/s-95.2MiB/s (99.8MB/s-99.8MB/s), io=667MiB (699MB), run=7001-7001msec

$ rm /tmp/fio-output_512_2_io_uring_cmd-bdev_xnvme_2.txt
rm: cannot remove '/tmp/fio-output_512_2_io_uring_cmd-bdev_xnvme_2.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring_cmd-bdev_xnvme" --rw="randread" --size="12G" --iodepth="2" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_2_io_uring_cmd-bdev_xnvme_2.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_512_2_io_uring_cmd-bdev_xnvme_2.txt
io_uring_cmd-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666789742,
  "timestamp_ms" : 1666789742679,
  "time" : "Wed Oct 26 15:09:02 2022",
  "jobs" : [
    {
      "jobname" : "io_uring_cmd-bdev_xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring_cmd-bdev_xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "2",
        "bs" : "512",
        "direct" : "0",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",
        "filename" : "Nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 808450048,
        "io_kbytes" : 789502,
        "bw_bytes" : 115476367,
        "bw" : 112769,
        "iops" : 225539.637195,
        "runtime" : 7001,
        "total_ios" : 1579003,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 672,
          "max" : 21169,
          "mean" : 953.871964,
          "stddev" : 122.852261,
          "N" : 1579003
        },
        "clat_ns" : {
          "min" : 752,
          "max" : 110307,
          "mean" : 7658.542619,
          "stddev" : 1855.562506,
          "N" : 1579004,
          "percentile" : {
            "1.000000" : 6624,
            "5.000000" : 6752,
            "10.000000" : 6816,
            "20.000000" : 6944,
            "30.000000" : 7072,
            "40.000000" : 7136,
            "50.000000" : 7200,
            "60.000000" : 7264,
            "70.000000" : 7392,
            "80.000000" : 7520,
            "90.000000" : 9536,
            "95.000000" : 9792,
            "99.000000" : 17280,
            "99.500000" : 19328,
            "99.900000" : 20352,
            "99.950000" : 22912,
            "99.990000" : 62208
          }
        },
        "lat_ns" : {
          "min" : 7073,
          "max" : 111219,
          "mean" : 8612.414576,
          "stddev" : 1850.718560,
          "N" : 1579004
        },
        "bw_min" : 112339,
        "bw_max" : 113330,
        "bw_agg" : 100.000000,
        "bw_mean" : 112795.214286,
        "bw_dev" : 372.083361,
        "bw_samples" : 14,
        "iops_min" : 224678,
        "iops_max" : 226660,
        "iops_mean" : 225590.428571,
        "iops_stddev" : 744.166722,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 99.985714,
      "sys_cpu" : 0.000000,
      "ctx" : 12,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.296453,
        "2" : 99.703547,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.010000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010000,
        "10" : 96.579234,
        "20" : 3.241919,
        "50" : 0.156808,
        "100" : 0.021849,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 2,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring_cmd-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=7056: Wed Oct 26 15:09:02 2022
  read: IOPS=226k, BW=110MiB/s (115MB/s)(771MiB/7001msec)
    slat (nsec): min=672, max=21169, avg=953.87, stdev=122.85
    clat (nsec): min=752, max=110307, avg=7658.54, stdev=1855.56
     lat (usec): min=7, max=111, avg= 8.61, stdev= 1.85
    clat percentiles (nsec):
     |  1.00th=[ 6624],  5.00th=[ 6752], 10.00th=[ 6816], 20.00th=[ 6944],
     | 30.00th=[ 7072], 40.00th=[ 7136], 50.00th=[ 7200], 60.00th=[ 7264],
     | 70.00th=[ 7392], 80.00th=[ 7520], 90.00th=[ 9536], 95.00th=[ 9792],
     | 99.00th=[17280], 99.50th=[19328], 99.90th=[20352], 99.95th=[22912],
     | 99.99th=[62208]
   bw (  KiB/s): min=112339, max=113330, per=100.00%, avg=112795.21, stdev=372.08, samples=14
   iops        : min=224678, max=226660, avg=225590.43, stdev=744.17, samples=14
  lat (nsec)   : 1000=0.01%
  lat (usec)   : 4=0.01%, 10=96.58%, 20=3.24%, 50=0.16%, 100=0.02%
  lat (usec)   : 250=0.01%
  cpu          : usr=99.99%, sys=0.00%, ctx=12, majf=0, minf=0
  IO depths    : 1=0.3%, 2=99.7%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1579003,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=2

Run status group 0 (all jobs):
   READ: bw=110MiB/s (115MB/s), 110MiB/s-110MiB/s (115MB/s-115MB/s), io=771MiB (808MB), run=7001-7001msec

$ rm /tmp/fio-output_512_2_io_uring-reference_0.txt
rm: cannot remove '/tmp/fio-output_512_2_io_uring-reference_0.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring-reference" --rw="randread" --size="12G" --iodepth="2" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_2_io_uring-reference_0.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_2_io_uring-reference_0.txt
io_uring-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=io_uring, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666789756,
  "timestamp_ms" : 1666789756530,
  "time" : "Wed Oct 26 15:09:16 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-reference",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-reference",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "2",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "io_uring",
        "filename" : "/dev/nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 763407360,
        "io_kbytes" : 745515,
        "bw_bytes" : 109058194,
        "bw" : 106502,
        "iops" : 213004.142857,
        "runtime" : 7000,
        "total_ios" : 1491029,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1312,
          "max" : 15359,
          "mean" : 1507.511643,
          "stddev" : 157.070001,
          "N" : 1491029
        },
        "clat_ns" : {
          "min" : 1262,
          "max" : 111940,
          "mean" : 7604.344876,
          "stddev" : 2256.544363,
          "N" : 1491030,
          "percentile" : {
            "1.000000" : 5280,
            "5.000000" : 5344,
            "10.000000" : 5408,
            "20.000000" : 5472,
            "30.000000" : 5600,
            "40.000000" : 7264,
            "50.000000" : 7648,
            "60.000000" : 8032,
            "70.000000" : 9024,
            "80.000000" : 9152,
            "90.000000" : 9280,
            "95.000000" : 9408,
            "99.000000" : 16512,
            "99.500000" : 18560,
            "99.900000" : 23936,
            "99.950000" : 24192,
            "99.990000" : 62720
          }
        },
        "lat_ns" : {
          "min" : 5671,
          "max" : 113352,
          "mean" : 9111.856576,
          "stddev" : 2313.110543,
          "N" : 1491030
        },
        "bw_min" : 106106,
        "bw_max" : 106959,
        "bw_agg" : 100.000000,
        "bw_mean" : 106515.714286,
        "bw_dev" : 259.918457,
        "bw_samples" : 14,
        "iops_min" : 212212,
        "iops_max" : 213918,
        "iops_mean" : 213031.500000,
        "iops_stddev" : 519.943007,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 12.800000,
      "sys_cpu" : 50.100000,
      "ctx" : 749016,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 100.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.010000,
        "4" : 0.010000,
        "10" : 97.365913,
        "20" : 2.445761,
        "50" : 0.154725,
        "100" : 0.024144,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 2,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ],
  "disk_util" : [
    {
      "name" : "nvme0n1",
      "read_ios" : 2140162,
      "write_ios" : 0,
      "read_merges" : 0,
      "write_merges" : 0,
      "read_ticks" : 10972,
      "write_ticks" : 0,
      "in_queue" : 10972,
      "util" : 99.069123
    }
  ]
}

io_uring-reference: (groupid=0, jobs=1): err= 0: pid=7103: Wed Oct 26 15:09:16 2022
  read: IOPS=213k, BW=104MiB/s (109MB/s)(728MiB/7000msec)
    slat (nsec): min=1312, max=15359, avg=1507.51, stdev=157.07
    clat (nsec): min=1262, max=111940, avg=7604.34, stdev=2256.54
     lat (usec): min=5, max=113, avg= 9.11, stdev= 2.31
    clat percentiles (nsec):
     |  1.00th=[ 5280],  5.00th=[ 5344], 10.00th=[ 5408], 20.00th=[ 5472],
     | 30.00th=[ 5600], 40.00th=[ 7264], 50.00th=[ 7648], 60.00th=[ 8032],
     | 70.00th=[ 9024], 80.00th=[ 9152], 90.00th=[ 9280], 95.00th=[ 9408],
     | 99.00th=[16512], 99.50th=[18560], 99.90th=[23936], 99.95th=[24192],
     | 99.99th=[62720]
   bw (  KiB/s): min=106106, max=106959, per=100.00%, avg=106515.71, stdev=259.92, samples=14
   iops        : min=212212, max=213918, avg=213031.50, stdev=519.94, samples=14
  lat (usec)   : 2=0.01%, 4=0.01%, 10=97.37%, 20=2.45%, 50=0.15%
  lat (usec)   : 100=0.02%, 250=0.01%
  cpu          : usr=12.80%, sys=50.10%, ctx=749016, majf=0, minf=0
  IO depths    : 1=0.0%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1491029,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=2

Run status group 0 (all jobs):
   READ: bw=104MiB/s (109MB/s), 104MiB/s-104MiB/s (109MB/s-109MB/s), io=728MiB (763MB), run=7000-7000msec

Disk stats (read/write):
  nvme0n1: ios=2140162/0, merge=0/0, ticks=10972/0, in_queue=10972, util=99.07%

$ rm /tmp/fio-output_512_1_io_uring_cmd-xnvme_1.txt
rm: cannot remove '/tmp/fio-output_512_1_io_uring_cmd-xnvme_1.txt': No such file or directory

$ rm /tmp/fio-output_512_2_io_uring-reference_1.txt
rm: cannot remove '/tmp/fio-output_512_2_io_uring-reference_1.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring-reference" --rw="randread" --size="12G" --iodepth="2" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_2_io_uring-reference_1.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_2_io_uring-reference_1.txt
io_uring-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=io_uring, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666789770,
  "timestamp_ms" : 1666789770326,
  "time" : "Wed Oct 26 15:09:30 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-reference",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-reference",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "2",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "io_uring",
        "filename" : "/dev/nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 757835776,
        "io_kbytes" : 740074,
        "bw_bytes" : 108246789,
        "bw" : 105709,
        "iops" : 211419.368662,
        "runtime" : 7001,
        "total_ios" : 1480147,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1332,
          "max" : 26660,
          "mean" : 1516.955765,
          "stddev" : 165.330542,
          "N" : 1480147
        },
        "clat_ns" : {
          "min" : 200,
          "max" : 109896,
          "mean" : 7661.495982,
          "stddev" : 2256.062018,
          "N" : 1480148,
          "percentile" : {
            "1.000000" : 5280,
            "5.000000" : 5408,
            "10.000000" : 5472,
            "20.000000" : 5536,
            "30.000000" : 5792,
            "40.000000" : 7328,
            "50.000000" : 7648,
            "60.000000" : 7968,
            "70.000000" : 9152,
            "80.000000" : 9152,
            "90.000000" : 9280,
            "95.000000" : 9536,
            "99.000000" : 16768,
            "99.500000" : 18560,
            "99.900000" : 23936,
            "99.950000" : 24448,
            "99.990000" : 65280
          }
        },
        "lat_ns" : {
          "min" : 5771,
          "max" : 111358,
          "mean" : 9178.451778,
          "stddev" : 2304.102186,
          "N" : 1480148
        },
        "bw_min" : 105505,
        "bw_max" : 106062,
        "bw_agg" : 100.000000,
        "bw_mean" : 105778.000000,
        "bw_dev" : 128.192764,
        "bw_samples" : 14,
        "iops_min" : 211010,
        "iops_max" : 212124,
        "iops_mean" : 211556.142857,
        "iops_stddev" : 256.392385,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 13.585714,
      "sys_cpu" : 49.757143,
      "ctx" : 743789,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 100.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010000,
        "10" : 97.308105,
        "20" : 2.463809,
        "50" : 0.192211,
        "100" : 0.026146,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 2,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ],
  "disk_util" : [
    {
      "name" : "nvme0n1",
      "read_ios" : 2124569,
      "write_ios" : 0,
      "read_merges" : 0,
      "write_merges" : 0,
      "read_ticks" : 10947,
      "write_ticks" : 0,
      "in_queue" : 10947,
      "util" : 99.068754
    }
  ]
}

io_uring-reference: (groupid=0, jobs=1): err= 0: pid=7150: Wed Oct 26 15:09:30 2022
  read: IOPS=211k, BW=103MiB/s (108MB/s)(723MiB/7001msec)
    slat (nsec): min=1332, max=26660, avg=1516.96, stdev=165.33
    clat (nsec): min=200, max=109896, avg=7661.50, stdev=2256.06
     lat (usec): min=5, max=111, avg= 9.18, stdev= 2.30
    clat percentiles (nsec):
     |  1.00th=[ 5280],  5.00th=[ 5408], 10.00th=[ 5472], 20.00th=[ 5536],
     | 30.00th=[ 5792], 40.00th=[ 7328], 50.00th=[ 7648], 60.00th=[ 7968],
     | 70.00th=[ 9152], 80.00th=[ 9152], 90.00th=[ 9280], 95.00th=[ 9536],
     | 99.00th=[16768], 99.50th=[18560], 99.90th=[23936], 99.95th=[24448],
     | 99.99th=[65280]
   bw (  KiB/s): min=105505, max=106062, per=100.00%, avg=105778.00, stdev=128.19, samples=14
   iops        : min=211010, max=212124, avg=211556.14, stdev=256.39, samples=14
  lat (nsec)   : 250=0.01%
  lat (usec)   : 4=0.01%, 10=97.31%, 20=2.46%, 50=0.19%, 100=0.03%
  lat (usec)   : 250=0.01%
  cpu          : usr=13.59%, sys=49.76%, ctx=743789, majf=0, minf=0
  IO depths    : 1=0.0%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1480147,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=2

Run status group 0 (all jobs):
   READ: bw=103MiB/s (108MB/s), 103MiB/s-103MiB/s (108MB/s-108MB/s), io=723MiB (758MB), run=7001-7001msec

Disk stats (read/write):
  nvme0n1: ios=2124569/0, merge=0/0, ticks=10947/0, in_queue=10947, util=99.07%

$ rm /tmp/fio-output_512_2_io_uring-reference_2.txt
rm: cannot remove '/tmp/fio-output_512_2_io_uring-reference_2.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring-reference" --rw="randread" --size="12G" --iodepth="2" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_2_io_uring-reference_2.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_2_io_uring-reference_2.txt
io_uring-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=io_uring, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666789784,
  "timestamp_ms" : 1666789784103,
  "time" : "Wed Oct 26 15:09:44 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-reference",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-reference",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "2",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "io_uring",
        "filename" : "/dev/nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 760478720,
        "io_kbytes" : 742655,
        "bw_bytes" : 108624299,
        "bw" : 106078,
        "iops" : 212156.691901,
        "runtime" : 7001,
        "total_ios" : 1485309,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1332,
          "max" : 22673,
          "mean" : 1514.181856,
          "stddev" : 153.990339,
          "N" : 1485309
        },
        "clat_ns" : {
          "min" : 100,
          "max" : 117761,
          "mean" : 7616.194574,
          "stddev" : 2211.320112,
          "N" : 1485310,
          "percentile" : {
            "1.000000" : 5280,
            "5.000000" : 5344,
            "10.000000" : 5344,
            "20.000000" : 5472,
            "30.000000" : 5664,
            "40.000000" : 7328,
            "50.000000" : 7648,
            "60.000000" : 7968,
            "70.000000" : 9024,
            "80.000000" : 9152,
            "90.000000" : 9280,
            "95.000000" : 9408,
            "99.000000" : 16512,
            "99.500000" : 18560,
            "99.900000" : 23680,
            "99.950000" : 24192,
            "99.990000" : 59648
          }
        },
        "lat_ns" : {
          "min" : 5700,
          "max" : 119204,
          "mean" : 9130.376490,
          "stddev" : 2257.397056,
          "N" : 1485310
        },
        "bw_min" : 105355,
        "bw_max" : 106441,
        "bw_agg" : 100.000000,
        "bw_mean" : 106110.214286,
        "bw_dev" : 316.062598,
        "bw_samples" : 14,
        "iops_min" : 210711,
        "iops_max" : 212882,
        "iops_mean" : 212220.571429,
        "iops_stddev" : 631.966980,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 13.228571,
      "sys_cpu" : 50.228571,
      "ctx" : 746364,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 100.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010000,
        "10" : 97.327155,
        "20" : 2.466490,
        "50" : 0.177135,
        "100" : 0.019592,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 2,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ],
  "disk_util" : [
    {
      "name" : "nvme0n1",
      "read_ios" : 2126963,
      "write_ios" : 0,
      "read_merges" : 0,
      "write_merges" : 0,
      "read_ticks" : 10923,
      "write_ticks" : 0,
      "in_queue" : 10923,
      "util" : 99.078934
    }
  ]
}

io_uring-reference: (groupid=0, jobs=1): err= 0: pid=7196: Wed Oct 26 15:09:44 2022
  read: IOPS=212k, BW=104MiB/s (109MB/s)(725MiB/7001msec)
    slat (nsec): min=1332, max=22673, avg=1514.18, stdev=153.99
    clat (nsec): min=100, max=117761, avg=7616.19, stdev=2211.32
     lat (usec): min=5, max=119, avg= 9.13, stdev= 2.26
    clat percentiles (nsec):
     |  1.00th=[ 5280],  5.00th=[ 5344], 10.00th=[ 5344], 20.00th=[ 5472],
     | 30.00th=[ 5664], 40.00th=[ 7328], 50.00th=[ 7648], 60.00th=[ 7968],
     | 70.00th=[ 9024], 80.00th=[ 9152], 90.00th=[ 9280], 95.00th=[ 9408],
     | 99.00th=[16512], 99.50th=[18560], 99.90th=[23680], 99.95th=[24192],
     | 99.99th=[59648]
   bw (  KiB/s): min=105355, max=106441, per=100.00%, avg=106110.21, stdev=316.06, samples=14
   iops        : min=210711, max=212882, avg=212220.57, stdev=631.97, samples=14
  lat (nsec)   : 250=0.01%
  lat (usec)   : 4=0.01%, 10=97.33%, 20=2.47%, 50=0.18%, 100=0.02%
  lat (usec)   : 250=0.01%
  cpu          : usr=13.23%, sys=50.23%, ctx=746364, majf=0, minf=0
  IO depths    : 1=0.0%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1485309,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=2

Run status group 0 (all jobs):
   READ: bw=104MiB/s (109MB/s), 104MiB/s-104MiB/s (109MB/s-109MB/s), io=725MiB (760MB), run=7001-7001msec

Disk stats (read/write):
  nvme0n1: ios=2126963/0, merge=0/0, ticks=10923/0, in_queue=10923, util=99.08%

$ rm /tmp/fio-output_512_2_io_uring-bdev_uring_0.txt
rm: cannot remove '/tmp/fio-output_512_2_io_uring-bdev_uring_0.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring-bdev_uring" --rw="randread" --size="12G" --iodepth="2" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_uring_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_2_io_uring-bdev_uring_0.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_512_2_io_uring-bdev_uring_0.txt
io_uring-bdev_uring: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666789799,
  "timestamp_ms" : 1666789799256,
  "time" : "Wed Oct 26 15:09:59 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-bdev_uring",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-bdev_uring",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "2",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_uring_foo.conf",
        "filename" : "Nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 928209408,
        "io_kbytes" : 906454,
        "bw_bytes" : 132582403,
        "bw" : 129475,
        "iops" : 258949.864305,
        "runtime" : 7001,
        "total_ios" : 1812908,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 110,
          "max" : 12924,
          "mean" : 173.489903,
          "stddev" : 32.365104,
          "N" : 1812908
        },
        "clat_ns" : {
          "min" : 5941,
          "max" : 118202,
          "mean" : 7276.558786,
          "stddev" : 1678.838912,
          "N" : 1812909,
          "percentile" : {
            "1.000000" : 6560,
            "5.000000" : 6688,
            "10.000000" : 6688,
            "20.000000" : 6752,
            "30.000000" : 6752,
            "40.000000" : 6816,
            "50.000000" : 6880,
            "60.000000" : 6880,
            "70.000000" : 6944,
            "80.000000" : 7200,
            "90.000000" : 8256,
            "95.000000" : 8384,
            "99.000000" : 14656,
            "99.500000" : 18560,
            "99.900000" : 20352,
            "99.950000" : 21120,
            "99.990000" : 62720
          }
        },
        "lat_ns" : {
          "min" : 6111,
          "max" : 118372,
          "mean" : 7450.048687,
          "stddev" : 1675.358061,
          "N" : 1812909
        },
        "bw_min" : 128981,
        "bw_max" : 130259,
        "bw_agg" : 100.000000,
        "bw_mean" : 129499.928571,
        "bw_dev" : 474.433989,
        "bw_samples" : 14,
        "iops_min" : 257963,
        "iops_max" : 260518,
        "iops_mean" : 258999.928571,
        "iops_stddev" : 948.783875,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 32.957143,
      "sys_cpu" : 67.014286,
      "ctx" : 11,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 8.368213,
        "2" : 91.631787,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 97.055173,
        "20" : 2.794626,
        "50" : 0.126316,
        "100" : 0.023608,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 2,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring-bdev_uring: (groupid=0, jobs=1): err= 0: pid=7252: Wed Oct 26 15:09:59 2022
  read: IOPS=259k, BW=126MiB/s (133MB/s)(885MiB/7001msec)
    slat (nsec): min=110, max=12924, avg=173.49, stdev=32.37
    clat (usec): min=5, max=118, avg= 7.28, stdev= 1.68
     lat (usec): min=6, max=118, avg= 7.45, stdev= 1.68
    clat percentiles (nsec):
     |  1.00th=[ 6560],  5.00th=[ 6688], 10.00th=[ 6688], 20.00th=[ 6752],
     | 30.00th=[ 6752], 40.00th=[ 6816], 50.00th=[ 6880], 60.00th=[ 6880],
     | 70.00th=[ 6944], 80.00th=[ 7200], 90.00th=[ 8256], 95.00th=[ 8384],
     | 99.00th=[14656], 99.50th=[18560], 99.90th=[20352], 99.95th=[21120],
     | 99.99th=[62720]
   bw (  KiB/s): min=128981, max=130259, per=100.00%, avg=129499.93, stdev=474.43, samples=14
   iops        : min=257963, max=260518, avg=258999.93, stdev=948.78, samples=14
  lat (usec)   : 10=97.06%, 20=2.79%, 50=0.13%, 100=0.02%, 250=0.01%
  cpu          : usr=32.96%, sys=67.01%, ctx=11, majf=0, minf=0
  IO depths    : 1=8.4%, 2=91.6%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1812908,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=2

Run status group 0 (all jobs):
   READ: bw=126MiB/s (133MB/s), 126MiB/s-126MiB/s (133MB/s-133MB/s), io=885MiB (928MB), run=7001-7001msec

$ rm /tmp/fio-output_512_2_io_uring-bdev_uring_1.txt
rm: cannot remove '/tmp/fio-output_512_2_io_uring-bdev_uring_1.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring_cmd-xnvme" --rw="randread" --size="12G" --iodepth="1" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring_cmd" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_512_1_io_uring_cmd-xnvme_1.txt" --cpus_allowed="1"
$ /opt/fio/bin/fio --name="io_uring-bdev_uring" --rw="randread" --size="12G" --iodepth="2" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_uring_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_2_io_uring-bdev_uring_1.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_512_2_io_uring-bdev_uring_1.txt
io_uring-bdev_uring: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666789814,
  "timestamp_ms" : 1666789814565,
  "time" : "Wed Oct 26 15:10:14 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-bdev_uring",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-bdev_uring",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "2",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_uring_foo.conf",
        "filename" : "Nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 919181824,
        "io_kbytes" : 897638,
        "bw_bytes" : 131292933,
        "bw" : 128215,
        "iops" : 256431.366948,
        "runtime" : 7001,
        "total_ios" : 1795276,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 110,
          "max" : 11972,
          "mean" : 171.000074,
          "stddev" : 44.899254,
          "N" : 1795276
        },
        "clat_ns" : {
          "min" : 5971,
          "max" : 119064,
          "mean" : 7345.391732,
          "stddev" : 1661.364011,
          "N" : 1795277,
          "percentile" : {
            "1.000000" : 6624,
            "5.000000" : 6688,
            "10.000000" : 6752,
            "20.000000" : 6816,
            "30.000000" : 6816,
            "40.000000" : 6880,
            "50.000000" : 6880,
            "60.000000" : 6944,
            "70.000000" : 7008,
            "80.000000" : 7840,
            "90.000000" : 8256,
            "95.000000" : 8512,
            "99.000000" : 14784,
            "99.500000" : 18560,
            "99.900000" : 20096,
            "99.950000" : 20864,
            "99.990000" : 62720
          }
        },
        "lat_ns" : {
          "min" : 6151,
          "max" : 119244,
          "mean" : 7516.391811,
          "stddev" : 1658.455431,
          "N" : 1795277
        },
        "bw_min" : 127602,
        "bw_max" : 128753,
        "bw_agg" : 100.000000,
        "bw_mean" : 128239.071429,
        "bw_dev" : 367.154225,
        "bw_samples" : 14,
        "iops_min" : 255204,
        "iops_max" : 257506,
        "iops_mean" : 256478.214286,
        "iops_stddev" : 734.226771,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 33.300000,
      "sys_cpu" : 66.671429,
      "ctx" : 10,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 9.490184,
        "2" : 90.509816,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 97.017896,
        "20" : 2.866412,
        "50" : 0.094359,
        "100" : 0.021055,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 2,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring-bdev_uring: (groupid=0, jobs=1): err= 0: pid=7309: Wed Oct 26 15:10:14 2022
  read: IOPS=256k, BW=125MiB/s (131MB/s)(877MiB/7001msec)
    slat (nsec): min=110, max=11972, avg=171.00, stdev=44.90
    clat (usec): min=5, max=119, avg= 7.35, stdev= 1.66
     lat (usec): min=6, max=119, avg= 7.52, stdev= 1.66
    clat percentiles (nsec):
     |  1.00th=[ 6624],  5.00th=[ 6688], 10.00th=[ 6752], 20.00th=[ 6816],
     | 30.00th=[ 6816], 40.00th=[ 6880], 50.00th=[ 6880], 60.00th=[ 6944],
     | 70.00th=[ 7008], 80.00th=[ 7840], 90.00th=[ 8256], 95.00th=[ 8512],
     | 99.00th=[14784], 99.50th=[18560], 99.90th=[20096], 99.95th=[20864],
     | 99.99th=[62720]
   bw (  KiB/s): min=127602, max=128753, per=100.00%, avg=128239.07, stdev=367.15, samples=14
   iops        : min=255204, max=257506, avg=256478.21, stdev=734.23, samples=14
  lat (usec)   : 10=97.02%, 20=2.87%, 50=0.09%, 100=0.02%, 250=0.01%
  cpu          : usr=33.30%, sys=66.67%, ctx=10, majf=0, minf=0
  IO depths    : 1=9.5%, 2=90.5%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1795276,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=2

Run status group 0 (all jobs):
   READ: bw=125MiB/s (131MB/s), 125MiB/s-125MiB/s (131MB/s-131MB/s), io=877MiB (919MB), run=7001-7001msec

$ rm /tmp/fio-output_512_2_io_uring-bdev_uring_2.txt
rm: cannot remove '/tmp/fio-output_512_2_io_uring-bdev_uring_2.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring-bdev_uring" --rw="randread" --size="12G" --iodepth="2" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_uring_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_2_io_uring-bdev_uring_2.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_512_2_io_uring-bdev_uring_2.txt
io_uring-bdev_uring: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666789829,
  "timestamp_ms" : 1666789829766,
  "time" : "Wed Oct 26 15:10:29 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-bdev_uring",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-bdev_uring",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "2",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_uring_foo.conf",
        "filename" : "Nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 924888576,
        "io_kbytes" : 903211,
        "bw_bytes" : 132108066,
        "bw" : 129011,
        "iops" : 258023.425225,
        "runtime" : 7001,
        "total_ios" : 1806422,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 110,
          "max" : 11772,
          "mean" : 169.994528,
          "stddev" : 41.933383,
          "N" : 1806422
        },
        "clat_ns" : {
          "min" : 5952,
          "max" : 102713,
          "mean" : 7298.990791,
          "stddev" : 1647.972135,
          "N" : 1806422,
          "percentile" : {
            "1.000000" : 6624,
            "5.000000" : 6688,
            "10.000000" : 6752,
            "20.000000" : 6752,
            "30.000000" : 6816,
            "40.000000" : 6816,
            "50.000000" : 6880,
            "60.000000" : 6880,
            "70.000000" : 6944,
            "80.000000" : 7264,
            "90.000000" : 8256,
            "95.000000" : 8384,
            "99.000000" : 14784,
            "99.500000" : 18560,
            "99.900000" : 20352,
            "99.950000" : 20864,
            "99.990000" : 62208
          }
        },
        "lat_ns" : {
          "min" : 6122,
          "max" : 102883,
          "mean" : 7468.985319,
          "stddev" : 1644.499905,
          "N" : 1806422
        },
        "bw_min" : 125789,
        "bw_max" : 129837,
        "bw_agg" : 100.000000,
        "bw_mean" : 129033.714286,
        "bw_dev" : 1030.428994,
        "bw_samples" : 14,
        "iops_min" : 251579,
        "iops_max" : 259674,
        "iops_mean" : 258067.500000,
        "iops_stddev" : 2060.615769,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 36.842857,
      "sys_cpu" : 63.128571,
      "ctx" : 16,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 8.782610,
        "2" : 91.217390,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 97.043991,
        "20" : 2.821212,
        "50" : 0.114259,
        "100" : 0.020427,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 2,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring-bdev_uring: (groupid=0, jobs=1): err= 0: pid=7367: Wed Oct 26 15:10:29 2022
  read: IOPS=258k, BW=126MiB/s (132MB/s)(882MiB/7001msec)
    slat (nsec): min=110, max=11772, avg=169.99, stdev=41.93
    clat (usec): min=5, max=102, avg= 7.30, stdev= 1.65
     lat (usec): min=6, max=102, avg= 7.47, stdev= 1.64
    clat percentiles (nsec):
     |  1.00th=[ 6624],  5.00th=[ 6688], 10.00th=[ 6752], 20.00th=[ 6752],
     | 30.00th=[ 6816], 40.00th=[ 6816], 50.00th=[ 6880], 60.00th=[ 6880],
     | 70.00th=[ 6944], 80.00th=[ 7264], 90.00th=[ 8256], 95.00th=[ 8384],
     | 99.00th=[14784], 99.50th=[18560], 99.90th=[20352], 99.95th=[20864],
     | 99.99th=[62208]
   bw (  KiB/s): min=125789, max=129837, per=100.00%, avg=129033.71, stdev=1030.43, samples=14
   iops        : min=251579, max=259674, avg=258067.50, stdev=2060.62, samples=14
  lat (usec)   : 10=97.04%, 20=2.82%, 50=0.11%, 100=0.02%, 250=0.01%
  cpu          : usr=36.84%, sys=63.13%, ctx=16, majf=0, minf=0
  IO depths    : 1=8.8%, 2=91.2%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1806422,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=2

Run status group 0 (all jobs):
   READ: bw=126MiB/s (132MB/s), 126MiB/s-126MiB/s (132MB/s-132MB/s), io=882MiB (925MB), run=7001-7001msec

$ rm /tmp/fio-output_512_2_io_uring-bdev_xnvme_0.txt
rm: cannot remove '/tmp/fio-output_512_2_io_uring-bdev_xnvme_0.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring-bdev_xnvme" --rw="randread" --size="12G" --iodepth="2" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_2_io_uring-bdev_xnvme_0.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_512_2_io_uring-bdev_xnvme_0.txt
io_uring-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666789845,
  "timestamp_ms" : 1666789845262,
  "time" : "Wed Oct 26 15:10:45 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-bdev_xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-bdev_xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "2",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring.conf",
        "filename" : "Nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 1071219200,
        "io_kbytes" : 1046112,
        "bw_bytes" : 153009455,
        "bw" : 149423,
        "iops" : 298846.307670,
        "runtime" : 7001,
        "total_ios" : 2092223,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1683,
          "max" : 35506,
          "mean" : 1886.948066,
          "stddev" : 122.911174,
          "N" : 2092224
        },
        "clat_ns" : {
          "min" : 260,
          "max" : 119775,
          "mean" : 4572.451873,
          "stddev" : 1788.671868,
          "N" : 2092224,
          "percentile" : {
            "1.000000" : 3888,
            "5.000000" : 4128,
            "10.000000" : 4192,
            "20.000000" : 4256,
            "30.000000" : 4256,
            "40.000000" : 4320,
            "50.000000" : 4320,
            "60.000000" : 4320,
            "70.000000" : 4384,
            "80.000000" : 4448,
            "90.000000" : 4448,
            "95.000000" : 4640,
            "99.000000" : 12608,
            "99.500000" : 17536,
            "99.900000" : 21888,
            "99.950000" : 22144,
            "99.990000" : 63232
          }
        },
        "lat_ns" : {
          "min" : 5139,
          "max" : 121608,
          "mean" : 6459.399939,
          "stddev" : 1792.133017,
          "N" : 2092224
        },
        "bw_min" : 148978,
        "bw_max" : 149936,
        "bw_agg" : 100.000000,
        "bw_mean" : 149450.785714,
        "bw_dev" : 305.810695,
        "bw_samples" : 14,
        "iops_min" : 297956,
        "iops_max" : 299873,
        "iops_mean" : 298901.642857,
        "iops_stddev" : 612.123116,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 44.157143,
      "sys_cpu" : 55.828571,
      "ctx" : 13,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 100.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.010000,
        "750" : 0.010000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 3.109707,
        "10" : 94.906518,
        "20" : 1.720562,
        "50" : 0.241657,
        "100" : 0.020982,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 2,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=7424: Wed Oct 26 15:10:45 2022
  read: IOPS=299k, BW=146MiB/s (153MB/s)(1022MiB/7001msec)
    slat (nsec): min=1683, max=35506, avg=1886.95, stdev=122.91
    clat (nsec): min=260, max=119775, avg=4572.45, stdev=1788.67
     lat (usec): min=5, max=121, avg= 6.46, stdev= 1.79
    clat percentiles (nsec):
     |  1.00th=[ 3888],  5.00th=[ 4128], 10.00th=[ 4192], 20.00th=[ 4256],
     | 30.00th=[ 4256], 40.00th=[ 4320], 50.00th=[ 4320], 60.00th=[ 4320],
     | 70.00th=[ 4384], 80.00th=[ 4448], 90.00th=[ 4448], 95.00th=[ 4640],
     | 99.00th=[12608], 99.50th=[17536], 99.90th=[21888], 99.95th=[22144],
     | 99.99th=[63232]
   bw (  KiB/s): min=148978, max=149936, per=100.00%, avg=149450.79, stdev=305.81, samples=14
   iops        : min=297956, max=299873, avg=298901.64, stdev=612.12, samples=14
  lat (nsec)   : 500=0.01%, 750=0.01%
  lat (usec)   : 4=3.11%, 10=94.91%, 20=1.72%, 50=0.24%, 100=0.02%
  lat (usec)   : 250=0.01%
  cpu          : usr=44.16%, sys=55.83%, ctx=13, majf=0, minf=0
  IO depths    : 1=0.0%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=2092223,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=2

Run status group 0 (all jobs):
   READ: bw=146MiB/s (153MB/s), 146MiB/s-146MiB/s (153MB/s-153MB/s), io=1022MiB (1071MB), run=7001-7001msec

$ rm /tmp/fio-output_512_2_io_uring-bdev_xnvme_1.txt
rm: cannot remove '/tmp/fio-output_512_2_io_uring-bdev_xnvme_1.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring-bdev_xnvme" --rw="randread" --size="12G" --iodepth="2" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_2_io_uring-bdev_xnvme_1.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_512_1_io_uring_cmd-xnvme_1.txt
io_uring_cmd-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666789203,
  "timestamp_ms" : 1666789203700,
  "time" : "Wed Oct 26 15:00:03 2022",
  "jobs" : [
    {
      "jobname" : "io_uring_cmd-xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring_cmd-xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "1",
        "bs" : "512",
        "direct" : "0",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
        "xnvme_be" : "linux",
        "xnvme_async" : "io_uring_cmd",
        "xnvme_sync" : "nvme",
        "xnvme_admin" : "nvme",
        "xnvme_dev_nsid" : "1",
        "filename" : "/dev/ng0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 394408960,
        "io_kbytes" : 385165,
        "bw_bytes" : 56336089,
        "bw" : 55015,
        "iops" : 110031.424082,
        "runtime" : 7001,
        "total_ios" : 770330,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1402,
          "max" : 14557,
          "mean" : 1512.964761,
          "stddev" : 62.706972,
          "N" : 770330
        },
        "clat_ns" : {
          "min" : 260,
          "max" : 97473,
          "mean" : 7280.894260,
          "stddev" : 982.896349,
          "N" : 770330,
          "percentile" : {
            "1.000000" : 6752,
            "5.000000" : 7008,
            "10.000000" : 7072,
            "20.000000" : 7072,
            "30.000000" : 7072,
            "40.000000" : 7136,
            "50.000000" : 7200,
            "60.000000" : 7200,
            "70.000000" : 7264,
            "80.000000" : 7264,
            "90.000000" : 7392,
            "95.000000" : 7456,
            "99.000000" : 10432,
            "99.500000" : 13632,
            "99.900000" : 15680,
            "99.950000" : 15808,
            "99.990000" : 58112
          }
        },
        "lat_ns" : {
          "min" : 6993,
          "max" : 98975,
          "mean" : 8793.859020,
          "stddev" : 985.772717,
          "N" : 770330
        },
        "bw_min" : 54788,
        "bw_max" : 55269,
        "bw_agg" : 100.000000,
        "bw_mean" : 55025.500000,
        "bw_dev" : 130.894408,
        "bw_samples" : 14,
        "iops_min" : 109576,
        "iops_max" : 110538,
        "iops_mean" : 110051.000000,
        "iops_stddev" : 261.788817,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 8.414286,
      "sys_cpu" : 35.785714,
      "ctx" : 770332,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 100.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.010000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.010000,
        "4" : 0.010000,
        "10" : 98.690691,
        "20" : 1.290486,
        "50" : 0.010000,
        "100" : 0.014799,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 1,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring_cmd-xnvme: (groupid=0, jobs=1): err= 0: pid=5122: Wed Oct 26 15:00:03 2022
  read: IOPS=110k, BW=53.7MiB/s (56.3MB/s)(376MiB/7001msec)
    slat (nsec): min=1402, max=14557, avg=1512.96, stdev=62.71
    clat (nsec): min=260, max=97473, avg=7280.89, stdev=982.90
     lat (nsec): min=6993, max=98975, avg=8793.86, stdev=985.77
    clat percentiles (nsec):
     |  1.00th=[ 6752],  5.00th=[ 7008], 10.00th=[ 7072], 20.00th=[ 7072],
     | 30.00th=[ 7072], 40.00th=[ 7136], 50.00th=[ 7200], 60.00th=[ 7200],
     | 70.00th=[ 7264], 80.00th=[ 7264], 90.00th=[ 7392], 95.00th=[ 7456],
     | 99.00th=[10432], 99.50th=[13632], 99.90th=[15680], 99.95th=[15808],
     | 99.99th=[58112]
   bw (  KiB/s): min=54788, max=55269, per=100.00%, avg=55025.50, stdev=130.89, samples=14
   iops        : min=109576, max=110538, avg=110051.00, stdev=261.79, samples=14
  lat (nsec)   : 500=0.01%
  lat (usec)   : 2=0.01%, 4=0.01%, 10=98.69%, 20=1.29%, 50=0.01%
  lat (usec)   : 100=0.01%
  cpu          : usr=8.41%, sys=35.79%, ctx=770332, majf=0, minf=0
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=770330,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=53.7MiB/s (56.3MB/s), 53.7MiB/s-53.7MiB/s (56.3MB/s-56.3MB/s), io=376MiB (394MB), run=7001-7001msec

$ cat /tmp/fio-output_512_2_io_uring-bdev_xnvme_1.txt
io_uring-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666789860,
  "timestamp_ms" : 1666789860552,
  "time" : "Wed Oct 26 15:11:00 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-bdev_xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-bdev_xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "2",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring.conf",
        "filename" : "Nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 1074551808,
        "io_kbytes" : 1049367,
        "bw_bytes" : 153485474,
        "bw" : 149888,
        "iops" : 299776.174832,
        "runtime" : 7001,
        "total_ios" : 2098733,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1673,
          "max" : 23223,
          "mean" : 1867.469730,
          "stddev" : 122.869433,
          "N" : 2098733
        },
        "clat_ns" : {
          "min" : 230,
          "max" : 115426,
          "mean" : 4579.960121,
          "stddev" : 1762.455468,
          "N" : 2098734,
          "percentile" : {
            "1.000000" : 3888,
            "5.000000" : 4128,
            "10.000000" : 4192,
            "20.000000" : 4256,
            "30.000000" : 4256,
            "40.000000" : 4320,
            "50.000000" : 4320,
            "60.000000" : 4384,
            "70.000000" : 4384,
            "80.000000" : 4448,
            "90.000000" : 4512,
            "95.000000" : 4640,
            "99.000000" : 12608,
            "99.500000" : 17536,
            "99.900000" : 21888,
            "99.950000" : 22400,
            "99.990000" : 61184
          }
        },
        "lat_ns" : {
          "min" : 5070,
          "max" : 117269,
          "mean" : 6447.429815,
          "stddev" : 1765.832177,
          "N" : 2098734
        },
        "bw_min" : 149368,
        "bw_max" : 150878,
        "bw_agg" : 100.000000,
        "bw_mean" : 149916.857143,
        "bw_dev" : 485.029080,
        "bw_samples" : 14,
        "iops_min" : 298736,
        "iops_max" : 301757,
        "iops_mean" : 299833.785714,
        "iops_stddev" : 970.210618,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 28.900000,
      "sys_cpu" : 71.085714,
      "ctx" : 9,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 100.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.010000,
        "500" : 0.010000,
        "750" : 0.000000,
        "1000" : 0.010000
      },
      "latency_us" : {
        "2" : 0.010000,
        "4" : 3.019584,
        "10" : 94.994837,
        "20" : 1.723421,
        "50" : 0.241717,
        "100" : 0.019631,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 2,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=7482: Wed Oct 26 15:11:00 2022
  read: IOPS=300k, BW=146MiB/s (153MB/s)(1025MiB/7001msec)
    slat (nsec): min=1673, max=23223, avg=1867.47, stdev=122.87
    clat (nsec): min=230, max=115426, avg=4579.96, stdev=1762.46
     lat (usec): min=5, max=117, avg= 6.45, stdev= 1.77
    clat percentiles (nsec):
     |  1.00th=[ 3888],  5.00th=[ 4128], 10.00th=[ 4192], 20.00th=[ 4256],
     | 30.00th=[ 4256], 40.00th=[ 4320], 50.00th=[ 4320], 60.00th=[ 4384],
     | 70.00th=[ 4384], 80.00th=[ 4448], 90.00th=[ 4512], 95.00th=[ 4640],
     | 99.00th=[12608], 99.50th=[17536], 99.90th=[21888], 99.95th=[22400],
     | 99.99th=[61184]
   bw (  KiB/s): min=149368, max=150878, per=100.00%, avg=149916.86, stdev=485.03, samples=14
   iops        : min=298736, max=301757, avg=299833.79, stdev=970.21, samples=14
  lat (nsec)   : 250=0.01%, 500=0.01%, 1000=0.01%
  lat (usec)   : 2=0.01%, 4=3.02%, 10=94.99%, 20=1.72%, 50=0.24%
  lat (usec)   : 100=0.02%, 250=0.01%
  cpu          : usr=28.90%, sys=71.09%, ctx=9, majf=0, minf=0
  IO depths    : 1=0.0%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=2098733,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=2

Run status group 0 (all jobs):
   READ: bw=146MiB/s (153MB/s), 146MiB/s-146MiB/s (153MB/s-153MB/s), io=1025MiB (1075MB), run=7001-7001msec

$ rm /tmp/fio-output_512_2_io_uring-bdev_xnvme_2.txt
rm: cannot remove '/tmp/fio-output_512_2_io_uring-bdev_xnvme_2.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring-bdev_xnvme" --rw="randread" --size="12G" --iodepth="2" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_2_io_uring-bdev_xnvme_2.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_512_2_io_uring-bdev_xnvme_2.txt
io_uring-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666789875,
  "timestamp_ms" : 1666789875781,
  "time" : "Wed Oct 26 15:11:15 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-bdev_xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-bdev_xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "2",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring.conf",
        "filename" : "Nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 1068184064,
        "io_kbytes" : 1043148,
        "bw_bytes" : 152575926,
        "bw" : 148999,
        "iops" : 297999.714327,
        "runtime" : 7001,
        "total_ios" : 2086296,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1723,
          "max" : 23454,
          "mean" : 1919.903935,
          "stddev" : 128.714954,
          "N" : 2086296
        },
        "clat_ns" : {
          "min" : 250,
          "max" : 98895,
          "mean" : 4553.953938,
          "stddev" : 1784.346449,
          "N" : 2086297,
          "percentile" : {
            "1.000000" : 3856,
            "5.000000" : 4128,
            "10.000000" : 4192,
            "20.000000" : 4192,
            "30.000000" : 4256,
            "40.000000" : 4256,
            "50.000000" : 4320,
            "60.000000" : 4320,
            "70.000000" : 4384,
            "80.000000" : 4384,
            "90.000000" : 4448,
            "95.000000" : 4640,
            "99.000000" : 12608,
            "99.500000" : 17536,
            "99.900000" : 21888,
            "99.950000" : 22144,
            "99.990000" : 62720
          }
        },
        "lat_ns" : {
          "min" : 5169,
          "max" : 100759,
          "mean" : 6473.857861,
          "stddev" : 1787.577180,
          "N" : 2086297
        },
        "bw_min" : 148578,
        "bw_max" : 149622,
        "bw_agg" : 100.000000,
        "bw_mean" : 149027.000000,
        "bw_dev" : 264.555069,
        "bw_samples" : 14,
        "iops_min" : 297156,
        "iops_max" : 299244,
        "iops_mean" : 298054.000000,
        "iops_stddev" : 529.110138,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 48.900000,
      "sys_cpu" : 51.085714,
      "ctx" : 15,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 100.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.010000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 3.277243,
        "10" : 94.723520,
        "20" : 1.731921,
        "50" : 0.245459,
        "100" : 0.021665,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 2,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=7539: Wed Oct 26 15:11:15 2022
  read: IOPS=298k, BW=146MiB/s (153MB/s)(1019MiB/7001msec)
    slat (nsec): min=1723, max=23454, avg=1919.90, stdev=128.71
    clat (nsec): min=250, max=98895, avg=4553.95, stdev=1784.35
     lat (usec): min=5, max=100, avg= 6.47, stdev= 1.79
    clat percentiles (nsec):
     |  1.00th=[ 3856],  5.00th=[ 4128], 10.00th=[ 4192], 20.00th=[ 4192],
     | 30.00th=[ 4256], 40.00th=[ 4256], 50.00th=[ 4320], 60.00th=[ 4320],
     | 70.00th=[ 4384], 80.00th=[ 4384], 90.00th=[ 4448], 95.00th=[ 4640],
     | 99.00th=[12608], 99.50th=[17536], 99.90th=[21888], 99.95th=[22144],
     | 99.99th=[62720]
   bw (  KiB/s): min=148578, max=149622, per=100.00%, avg=149027.00, stdev=264.56, samples=14
   iops        : min=297156, max=299244, avg=298054.00, stdev=529.11, samples=14
  lat (nsec)   : 500=0.01%
  lat (usec)   : 4=3.28%, 10=94.72%, 20=1.73%, 50=0.25%, 100=0.02%
  cpu          : usr=48.90%, sys=51.09%, ctx=15, majf=0, minf=0
  IO depths    : 1=0.0%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=2086296,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=2

Run status group 0 (all jobs):
   READ: bw=146MiB/s (153MB/s), 146MiB/s-146MiB/s (153MB/s-153MB/s), io=1019MiB (1068MB), run=7001-7001msec

$ rm /tmp/fio-output_512_2_io_uring-xnvme_0.txt
rm: cannot remove '/tmp/fio-output_512_2_io_uring-xnvme_0.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring-xnvme" --rw="randread" --size="12G" --iodepth="2" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_2_io_uring-xnvme_0.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_2_io_uring-xnvme_0.txt
io_uring-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666789889,
  "timestamp_ms" : 1666789889671,
  "time" : "Wed Oct 26 15:11:29 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "2",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
        "xnvme_be" : "linux",
        "xnvme_async" : "io_uring",
        "xnvme_sync" : "nvme",
        "xnvme_admin" : "nvme",
        "xnvme_dev_nsid" : "1",
        "filename" : "/dev/nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 755842560,
        "io_kbytes" : 738127,
        "bw_bytes" : 107962085,
        "bw" : 105431,
        "iops" : 210863.305242,
        "runtime" : 7001,
        "total_ios" : 1476254,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1373,
          "max" : 14708,
          "mean" : 1626.049088,
          "stddev" : 183.510630,
          "N" : 1476254
        },
        "clat_ns" : {
          "min" : 180,
          "max" : 118923,
          "mean" : 7596.831394,
          "stddev" : 2223.151074,
          "N" : 1476255,
          "percentile" : {
            "1.000000" : 5152,
            "5.000000" : 5280,
            "10.000000" : 5344,
            "20.000000" : 5472,
            "30.000000" : 6752,
            "40.000000" : 7264,
            "50.000000" : 7456,
            "60.000000" : 7712,
            "70.000000" : 8032,
            "80.000000" : 9280,
            "90.000000" : 9408,
            "95.000000" : 9536,
            "99.000000" : 16768,
            "99.500000" : 18560,
            "99.900000" : 23936,
            "99.950000" : 24448,
            "99.990000" : 61184
          }
        },
        "lat_ns" : {
          "min" : 5811,
          "max" : 120425,
          "mean" : 9222.880527,
          "stddev" : 2251.499860,
          "N" : 1476255
        },
        "bw_min" : 105209,
        "bw_max" : 105680,
        "bw_agg" : 100.000000,
        "bw_mean" : 105450.357143,
        "bw_dev" : 127.924020,
        "bw_samples" : 14,
        "iops_min" : 210418,
        "iops_max" : 211362,
        "iops_mean" : 210900.785714,
        "iops_stddev" : 255.828722,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 14.514286,
      "sys_cpu" : 51.542857,
      "ctx" : 741997,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 100.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010000,
        "10" : 97.311100,
        "20" : 2.439621,
        "50" : 0.215004,
        "100" : 0.024522,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 2,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring-xnvme: (groupid=0, jobs=1): err= 0: pid=7587: Wed Oct 26 15:11:29 2022
  read: IOPS=211k, BW=103MiB/s (108MB/s)(721MiB/7001msec)
    slat (nsec): min=1373, max=14708, avg=1626.05, stdev=183.51
    clat (nsec): min=180, max=118923, avg=7596.83, stdev=2223.15
     lat (usec): min=5, max=120, avg= 9.22, stdev= 2.25
    clat percentiles (nsec):
     |  1.00th=[ 5152],  5.00th=[ 5280], 10.00th=[ 5344], 20.00th=[ 5472],
     | 30.00th=[ 6752], 40.00th=[ 7264], 50.00th=[ 7456], 60.00th=[ 7712],
     | 70.00th=[ 8032], 80.00th=[ 9280], 90.00th=[ 9408], 95.00th=[ 9536],
     | 99.00th=[16768], 99.50th=[18560], 99.90th=[23936], 99.95th=[24448],
     | 99.99th=[61184]
   bw (  KiB/s): min=105209, max=105680, per=100.00%, avg=105450.36, stdev=127.92, samples=14
   iops        : min=210418, max=211362, avg=210900.79, stdev=255.83, samples=14
  lat (nsec)   : 250=0.01%
  lat (usec)   : 4=0.01%, 10=97.31%, 20=2.44%, 50=0.22%, 100=0.02%
  lat (usec)   : 250=0.01%
  cpu          : usr=14.51%, sys=51.54%, ctx=741997, majf=0, minf=0
  IO depths    : 1=0.0%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1476254,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=2

Run status group 0 (all jobs):
   READ: bw=103MiB/s (108MB/s), 103MiB/s-103MiB/s (108MB/s-108MB/s), io=721MiB (756MB), run=7001-7001msec

$ rm /tmp/fio-output_512_2_io_uring-xnvme_1.txt
rm: cannot remove '/tmp/fio-output_512_2_io_uring-xnvme_1.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring-xnvme" --rw="randread" --size="12G" --iodepth="2" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_2_io_uring-xnvme_1.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_2_io_uring-xnvme_1.txt
io_uring-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666789903,
  "timestamp_ms" : 1666789903447,
  "time" : "Wed Oct 26 15:11:43 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "2",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
        "xnvme_be" : "linux",
        "xnvme_async" : "io_uring",
        "xnvme_sync" : "nvme",
        "xnvme_admin" : "nvme",
        "xnvme_dev_nsid" : "1",
        "filename" : "/dev/nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 751161856,
        "io_kbytes" : 733556,
        "bw_bytes" : 107293508,
        "bw" : 104778,
        "iops" : 209557.491787,
        "runtime" : 7001,
        "total_ios" : 1467112,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1402,
          "max" : 30337,
          "mean" : 1675.573352,
          "stddev" : 230.937066,
          "N" : 1467112
        },
        "clat_ns" : {
          "min" : 2745,
          "max" : 119153,
          "mean" : 7591.607664,
          "stddev" : 2285.968158,
          "N" : 1467113,
          "percentile" : {
            "1.000000" : 5216,
            "5.000000" : 5280,
            "10.000000" : 5344,
            "20.000000" : 5472,
            "30.000000" : 5984,
            "40.000000" : 7136,
            "50.000000" : 7456,
            "60.000000" : 7712,
            "70.000000" : 8160,
            "80.000000" : 9408,
            "90.000000" : 9664,
            "95.000000" : 9792,
            "99.000000" : 16768,
            "99.500000" : 18560,
            "99.900000" : 24192,
            "99.950000" : 24448,
            "99.990000" : 64256
          }
        },
        "lat_ns" : {
          "min" : 5801,
          "max" : 120656,
          "mean" : 9267.180865,
          "stddev" : 2311.110610,
          "N" : 1467113
        },
        "bw_min" : 104352,
        "bw_max" : 105176,
        "bw_agg" : 100.000000,
        "bw_mean" : 104797.642857,
        "bw_dev" : 211.042689,
        "bw_samples" : 14,
        "iops_min" : 208705,
        "iops_max" : 210352,
        "iops_mean" : 209595.357143,
        "iops_stddev" : 421.922998,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 15.942857,
      "sys_cpu" : 51.414286,
      "ctx" : 737441,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 100.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010156,
        "10" : 97.018701,
        "20" : 2.717039,
        "50" : 0.231816,
        "100" : 0.022016,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 2,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring-xnvme: (groupid=0, jobs=1): err= 0: pid=7635: Wed Oct 26 15:11:43 2022
  read: IOPS=210k, BW=102MiB/s (107MB/s)(716MiB/7001msec)
    slat (nsec): min=1402, max=30337, avg=1675.57, stdev=230.94
    clat (usec): min=2, max=119, avg= 7.59, stdev= 2.29
     lat (usec): min=5, max=120, avg= 9.27, stdev= 2.31
    clat percentiles (nsec):
     |  1.00th=[ 5216],  5.00th=[ 5280], 10.00th=[ 5344], 20.00th=[ 5472],
     | 30.00th=[ 5984], 40.00th=[ 7136], 50.00th=[ 7456], 60.00th=[ 7712],
     | 70.00th=[ 8160], 80.00th=[ 9408], 90.00th=[ 9664], 95.00th=[ 9792],
     | 99.00th=[16768], 99.50th=[18560], 99.90th=[24192], 99.95th=[24448],
     | 99.99th=[64256]
   bw (  KiB/s): min=104352, max=105176, per=100.00%, avg=104797.64, stdev=211.04, samples=14
   iops        : min=208705, max=210352, avg=209595.36, stdev=421.92, samples=14
  lat (usec)   : 4=0.01%, 10=97.02%, 20=2.72%, 50=0.23%, 100=0.02%
  lat (usec)   : 250=0.01%
  cpu          : usr=15.94%, sys=51.41%, ctx=737441, majf=0, minf=0
  IO depths    : 1=0.0%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1467112,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=2

Run status group 0 (all jobs):
   READ: bw=102MiB/s (107MB/s), 102MiB/s-102MiB/s (107MB/s-107MB/s), io=716MiB (751MB), run=7001-7001msec

$ rm /tmp/fio-output_512_1_io_uring_cmd-xnvme_2.txt
rm: cannot remove '/tmp/fio-output_512_1_io_uring_cmd-xnvme_2.txt': No such file or directory

$ rm /tmp/fio-output_512_2_io_uring-xnvme_2.txt
rm: cannot remove '/tmp/fio-output_512_2_io_uring-xnvme_2.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring-xnvme" --rw="randread" --size="12G" --iodepth="2" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_2_io_uring-xnvme_2.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_2_io_uring-xnvme_2.txt
io_uring-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666789917,
  "timestamp_ms" : 1666789917192,
  "time" : "Wed Oct 26 15:11:57 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "2",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
        "xnvme_be" : "linux",
        "xnvme_async" : "io_uring",
        "xnvme_sync" : "nvme",
        "xnvme_admin" : "nvme",
        "xnvme_dev_nsid" : "1",
        "filename" : "/dev/nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 754146304,
        "io_kbytes" : 736471,
        "bw_bytes" : 107735186,
        "bw" : 105210,
        "iops" : 210420.142857,
        "runtime" : 7000,
        "total_ios" : 1472941,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1373,
          "max" : 39404,
          "mean" : 1621.156412,
          "stddev" : 192.150453,
          "N" : 1472941
        },
        "clat_ns" : {
          "min" : 180,
          "max" : 110587,
          "mean" : 7626.157111,
          "stddev" : 2228.400265,
          "N" : 1472942,
          "percentile" : {
            "1.000000" : 5216,
            "5.000000" : 5280,
            "10.000000" : 5344,
            "20.000000" : 5472,
            "30.000000" : 6752,
            "40.000000" : 7264,
            "50.000000" : 7456,
            "60.000000" : 7712,
            "70.000000" : 8096,
            "80.000000" : 9280,
            "90.000000" : 9408,
            "95.000000" : 9664,
            "99.000000" : 16768,
            "99.500000" : 18560,
            "99.900000" : 23936,
            "99.950000" : 24448,
            "99.990000" : 61696
          }
        },
        "lat_ns" : {
          "min" : 5741,
          "max" : 112090,
          "mean" : 9247.313388,
          "stddev" : 2260.379617,
          "N" : 1472942
        },
        "bw_min" : 104853,
        "bw_max" : 105510,
        "bw_agg" : 100.000000,
        "bw_mean" : 105237.769231,
        "bw_dev" : 207.918716,
        "bw_samples" : 13,
        "iops_min" : 209706,
        "iops_max" : 211020,
        "iops_mean" : 210475.769231,
        "iops_stddev" : 415.849162,
        "iops_samples" : 13
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 14.157143,
      "sys_cpu" : 51.885714,
      "ctx" : 740379,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 100.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010000,
        "10" : 97.277556,
        "20" : 2.467716,
        "50" : 0.220783,
        "100" : 0.024101,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 2,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring-xnvme: (groupid=0, jobs=1): err= 0: pid=7683: Wed Oct 26 15:11:57 2022
  read: IOPS=210k, BW=103MiB/s (108MB/s)(719MiB/7000msec)
    slat (nsec): min=1373, max=39404, avg=1621.16, stdev=192.15
    clat (nsec): min=180, max=110587, avg=7626.16, stdev=2228.40
     lat (usec): min=5, max=112, avg= 9.25, stdev= 2.26
    clat percentiles (nsec):
     |  1.00th=[ 5216],  5.00th=[ 5280], 10.00th=[ 5344], 20.00th=[ 5472],
     | 30.00th=[ 6752], 40.00th=[ 7264], 50.00th=[ 7456], 60.00th=[ 7712],
     | 70.00th=[ 8096], 80.00th=[ 9280], 90.00th=[ 9408], 95.00th=[ 9664],
     | 99.00th=[16768], 99.50th=[18560], 99.90th=[23936], 99.95th=[24448],
     | 99.99th=[61696]
   bw (  KiB/s): min=104853, max=105510, per=100.00%, avg=105237.77, stdev=207.92, samples=13
   iops        : min=209706, max=211020, avg=210475.77, stdev=415.85, samples=13
  lat (nsec)   : 250=0.01%
  lat (usec)   : 4=0.01%, 10=97.28%, 20=2.47%, 50=0.22%, 100=0.02%
  lat (usec)   : 250=0.01%
  cpu          : usr=14.16%, sys=51.89%, ctx=740379, majf=0, minf=0
  IO depths    : 1=0.0%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1472941,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=2

Run status group 0 (all jobs):
   READ: bw=103MiB/s (108MB/s), 103MiB/s-103MiB/s (108MB/s-108MB/s), io=719MiB (754MB), run=7000-7000msec

$ rm /tmp/fio-output_512_2_libaio-reference_0.txt
rm: cannot remove '/tmp/fio-output_512_2_libaio-reference_0.txt': No such file or directory

$ /opt/fio/bin/fio --name="libaio-reference" --rw="randread" --size="12G" --iodepth="2" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="libaio" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_2_libaio-reference_0.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_2_libaio-reference_0.txt
libaio-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=libaio, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666789931,
  "timestamp_ms" : 1666789931060,
  "time" : "Wed Oct 26 15:12:11 2022",
  "jobs" : [
    {
      "jobname" : "libaio-reference",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-reference",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "2",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "libaio",
        "filename" : "/dev/nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 799335936,
        "io_kbytes" : 780601,
        "bw_bytes" : 114174537,
        "bw" : 111498,
        "iops" : 222997.000429,
        "runtime" : 7001,
        "total_ios" : 1561202,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1442,
          "max" : 21661,
          "mean" : 1649.153308,
          "stddev" : 129.768110,
          "N" : 1561202
        },
        "clat_ns" : {
          "min" : 2645,
          "max" : 100528,
          "mean" : 7067.635010,
          "stddev" : 1738.217242,
          "N" : 1561203,
          "percentile" : {
            "1.000000" : 5856,
            "5.000000" : 5920,
            "10.000000" : 5984,
            "20.000000" : 6112,
            "30.000000" : 6880,
            "40.000000" : 6944,
            "50.000000" : 7008,
            "60.000000" : 7136,
            "70.000000" : 7200,
            "80.000000" : 7264,
            "90.000000" : 7392,
            "95.000000" : 7520,
            "99.000000" : 15936,
            "99.500000" : 18560,
            "99.900000" : 25216,
            "99.950000" : 25728,
            "99.990000" : 62720
          }
        },
        "lat_ns" : {
          "min" : 6422,
          "max" : 102221,
          "mean" : 8716.788250,
          "stddev" : 1739.886131,
          "N" : 1561203
        },
        "bw_min" : 110982,
        "bw_max" : 112688,
        "bw_agg" : 100.000000,
        "bw_mean" : 111528.500000,
        "bw_dev" : 424.069797,
        "bw_samples" : 14,
        "iops_min" : 221965,
        "iops_max" : 225376,
        "iops_mean" : 223057.071429,
        "iops_stddev" : 848.040498,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 20.328571,
      "sys_cpu" : 47.114286,
      "ctx" : 783978,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 100.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010000,
        "10" : 97.719770,
        "20" : 1.871763,
        "50" : 0.379771,
        "100" : 0.021009,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 2,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ],
  "disk_util" : [
    {
      "name" : "nvme0n1",
      "read_ios" : 2224647,
      "write_ios" : 0,
      "read_merges" : 0,
      "write_merges" : 0,
      "read_ticks" : 10878,
      "write_ticks" : 0,
      "in_queue" : 10878,
      "util" : 99.069123
    }
  ]
}

libaio-reference: (groupid=0, jobs=1): err= 0: pid=7730: Wed Oct 26 15:12:11 2022
  read: IOPS=223k, BW=109MiB/s (114MB/s)(762MiB/7001msec)
    slat (nsec): min=1442, max=21661, avg=1649.15, stdev=129.77
    clat (usec): min=2, max=100, avg= 7.07, stdev= 1.74
     lat (usec): min=6, max=102, avg= 8.72, stdev= 1.74
    clat percentiles (nsec):
     |  1.00th=[ 5856],  5.00th=[ 5920], 10.00th=[ 5984], 20.00th=[ 6112],
     | 30.00th=[ 6880], 40.00th=[ 6944], 50.00th=[ 7008], 60.00th=[ 7136],
     | 70.00th=[ 7200], 80.00th=[ 7264], 90.00th=[ 7392], 95.00th=[ 7520],
     | 99.00th=[15936], 99.50th=[18560], 99.90th=[25216], 99.95th=[25728],
     | 99.99th=[62720]
   bw (  KiB/s): min=110982, max=112688, per=100.00%, avg=111528.50, stdev=424.07, samples=14
   iops        : min=221965, max=225376, avg=223057.07, stdev=848.04, samples=14
  lat (usec)   : 4=0.01%, 10=97.72%, 20=1.87%, 50=0.38%, 100=0.02%
  lat (usec)   : 250=0.01%
  cpu          : usr=20.33%, sys=47.11%, ctx=783978, majf=0, minf=0
  IO depths    : 1=0.0%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1561202,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=2

Run status group 0 (all jobs):
   READ: bw=109MiB/s (114MB/s), 109MiB/s-109MiB/s (114MB/s-114MB/s), io=762MiB (799MB), run=7001-7001msec

Disk stats (read/write):
  nvme0n1: ios=2224647/0, merge=0/0, ticks=10878/0, in_queue=10878, util=99.07%

$ rm /tmp/fio-output_512_2_libaio-reference_1.txt
rm: cannot remove '/tmp/fio-output_512_2_libaio-reference_1.txt': No such file or directory

$ /opt/fio/bin/fio --name="libaio-reference" --rw="randread" --size="12G" --iodepth="2" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="libaio" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_2_libaio-reference_1.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_2_libaio-reference_1.txt
libaio-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=libaio, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666789944,
  "timestamp_ms" : 1666789944855,
  "time" : "Wed Oct 26 15:12:24 2022",
  "jobs" : [
    {
      "jobname" : "libaio-reference",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-reference",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "2",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "libaio",
        "filename" : "/dev/nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 815441920,
        "io_kbytes" : 796330,
        "bw_bytes" : 116475063,
        "bw" : 113745,
        "iops" : 227490.072847,
        "runtime" : 7001,
        "total_ios" : 1592658,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1472,
          "max" : 20098,
          "mean" : 1679.534212,
          "stddev" : 135.170532,
          "N" : 1592659
        },
        "clat_ns" : {
          "min" : 431,
          "max" : 91642,
          "mean" : 6852.600837,
          "stddev" : 1799.750306,
          "N" : 1592659,
          "percentile" : {
            "1.000000" : 5856,
            "5.000000" : 5920,
            "10.000000" : 5920,
            "20.000000" : 6048,
            "30.000000" : 6112,
            "40.000000" : 6240,
            "50.000000" : 6368,
            "60.000000" : 7008,
            "70.000000" : 7136,
            "80.000000" : 7328,
            "90.000000" : 7456,
            "95.000000" : 7584,
            "99.000000" : 15680,
            "99.500000" : 18560,
            "99.900000" : 25472,
            "99.950000" : 25984,
            "99.990000" : 62720
          }
        },
        "lat_ns" : {
          "min" : 5681,
          "max" : 93386,
          "mean" : 8532.135048,
          "stddev" : 1803.334792,
          "N" : 1592659
        },
        "bw_min" : 113374,
        "bw_max" : 114111,
        "bw_agg" : 100.000000,
        "bw_mean" : 113776.000000,
        "bw_dev" : 230.273082,
        "bw_samples" : 14,
        "iops_min" : 226748,
        "iops_max" : 228222,
        "iops_mean" : 227551.928571,
        "iops_stddev" : 460.425633,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 23.000000,
      "sys_cpu" : 47.142857,
      "ctx" : 800025,
      "majf" : 0,
      "minf" : 1,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 100.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.010000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010000,
        "10" : 97.734856,
        "20" : 1.887411,
        "50" : 0.346276,
        "100" : 0.023420,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 2,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ],
  "disk_util" : [
    {
      "name" : "nvme0n1",
      "read_ios" : 2265813,
      "write_ios" : 0,
      "read_merges" : 0,
      "write_merges" : 0,
      "read_ticks" : 11215,
      "write_ticks" : 0,
      "in_queue" : 11215,
      "util" : 99.069123
    }
  ]
}

libaio-reference: (groupid=0, jobs=1): err= 0: pid=7778: Wed Oct 26 15:12:24 2022
  read: IOPS=227k, BW=111MiB/s (116MB/s)(778MiB/7001msec)
    slat (nsec): min=1472, max=20098, avg=1679.53, stdev=135.17
    clat (nsec): min=431, max=91642, avg=6852.60, stdev=1799.75
     lat (nsec): min=5681, max=93386, avg=8532.14, stdev=1803.33
    clat percentiles (nsec):
     |  1.00th=[ 5856],  5.00th=[ 5920], 10.00th=[ 5920], 20.00th=[ 6048],
     | 30.00th=[ 6112], 40.00th=[ 6240], 50.00th=[ 6368], 60.00th=[ 7008],
     | 70.00th=[ 7136], 80.00th=[ 7328], 90.00th=[ 7456], 95.00th=[ 7584],
     | 99.00th=[15680], 99.50th=[18560], 99.90th=[25472], 99.95th=[25984],
     | 99.99th=[62720]
   bw (  KiB/s): min=113374, max=114111, per=100.00%, avg=113776.00, stdev=230.27, samples=14
   iops        : min=226748, max=228222, avg=227551.93, stdev=460.43, samples=14
  lat (nsec)   : 500=0.01%
  lat (usec)   : 4=0.01%, 10=97.73%, 20=1.89%, 50=0.35%, 100=0.02%
  cpu          : usr=23.00%, sys=47.14%, ctx=800025, majf=0, minf=1
  IO depths    : 1=0.0%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1592658,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=2

Run status group 0 (all jobs):
   READ: bw=111MiB/s (116MB/s), 111MiB/s-111MiB/s (116MB/s-116MB/s), io=778MiB (815MB), run=7001-7001msec

Disk stats (read/write):
  nvme0n1: ios=2265813/0, merge=0/0, ticks=11215/0, in_queue=11215, util=99.07%

$ rm /tmp/fio-output_512_2_libaio-reference_2.txt
rm: cannot remove '/tmp/fio-output_512_2_libaio-reference_2.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring_cmd-xnvme" --rw="randread" --size="12G" --iodepth="1" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring_cmd" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_512_1_io_uring_cmd-xnvme_2.txt" --cpus_allowed="1"
$ /opt/fio/bin/fio --name="libaio-reference" --rw="randread" --size="12G" --iodepth="2" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="libaio" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_2_libaio-reference_2.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_2_libaio-reference_2.txt
libaio-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=libaio, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666789958,
  "timestamp_ms" : 1666789958677,
  "time" : "Wed Oct 26 15:12:38 2022",
  "jobs" : [
    {
      "jobname" : "libaio-reference",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-reference",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "2",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "libaio",
        "filename" : "/dev/nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 812047872,
        "io_kbytes" : 793015,
        "bw_bytes" : 115990268,
        "bw" : 113271,
        "iops" : 226543.208113,
        "runtime" : 7001,
        "total_ios" : 1586029,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1452,
          "max" : 14798,
          "mean" : 1655.304542,
          "stddev" : 123.996712,
          "N" : 1586030
        },
        "clat_ns" : {
          "min" : 421,
          "max" : 120085,
          "mean" : 6916.411781,
          "stddev" : 1733.488478,
          "N" : 1586030,
          "percentile" : {
            "1.000000" : 5792,
            "5.000000" : 5856,
            "10.000000" : 5920,
            "20.000000" : 6048,
            "30.000000" : 6112,
            "40.000000" : 6816,
            "50.000000" : 6944,
            "60.000000" : 7008,
            "70.000000" : 7136,
            "80.000000" : 7264,
            "90.000000" : 7328,
            "95.000000" : 7520,
            "99.000000" : 15680,
            "99.500000" : 18560,
            "99.900000" : 24192,
            "99.950000" : 25728,
            "99.990000" : 61184
          }
        },
        "lat_ns" : {
          "min" : 5661,
          "max" : 121798,
          "mean" : 8571.716323,
          "stddev" : 1737.367517,
          "N" : 1586030
        },
        "bw_min" : 112211,
        "bw_max" : 114174,
        "bw_agg" : 100.000000,
        "bw_mean" : 113343.571429,
        "bw_dev" : 546.633292,
        "bw_samples" : 14,
        "iops_min" : 224422,
        "iops_max" : 228348,
        "iops_mean" : 226687.285714,
        "iops_stddev" : 1093.360904,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 21.314286,
      "sys_cpu" : 47.257143,
      "ctx" : 796376,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 100.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.010000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010000,
        "10" : 97.729802,
        "20" : 1.891264,
        "50" : 0.351885,
        "100" : 0.018789,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 2,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ],
  "disk_util" : [
    {
      "name" : "nvme0n1",
      "read_ios" : 2250040,
      "write_ios" : 0,
      "read_merges" : 0,
      "write_merges" : 0,
      "read_ticks" : 11049,
      "write_ticks" : 0,
      "in_queue" : 11049,
      "util" : 99.108381
    }
  ]
}

libaio-reference: (groupid=0, jobs=1): err= 0: pid=7824: Wed Oct 26 15:12:38 2022
  read: IOPS=227k, BW=111MiB/s (116MB/s)(774MiB/7001msec)
    slat (nsec): min=1452, max=14798, avg=1655.30, stdev=124.00
    clat (nsec): min=421, max=120085, avg=6916.41, stdev=1733.49
     lat (usec): min=5, max=121, avg= 8.57, stdev= 1.74
    clat percentiles (nsec):
     |  1.00th=[ 5792],  5.00th=[ 5856], 10.00th=[ 5920], 20.00th=[ 6048],
     | 30.00th=[ 6112], 40.00th=[ 6816], 50.00th=[ 6944], 60.00th=[ 7008],
     | 70.00th=[ 7136], 80.00th=[ 7264], 90.00th=[ 7328], 95.00th=[ 7520],
     | 99.00th=[15680], 99.50th=[18560], 99.90th=[24192], 99.95th=[25728],
     | 99.99th=[61184]
   bw (  KiB/s): min=112211, max=114174, per=100.00%, avg=113343.57, stdev=546.63, samples=14
   iops        : min=224422, max=228348, avg=226687.29, stdev=1093.36, samples=14
  lat (nsec)   : 500=0.01%
  lat (usec)   : 4=0.01%, 10=97.73%, 20=1.89%, 50=0.35%, 100=0.02%
  lat (usec)   : 250=0.01%
  cpu          : usr=21.31%, sys=47.26%, ctx=796376, majf=0, minf=0
  IO depths    : 1=0.0%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1586029,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=2

Run status group 0 (all jobs):
   READ: bw=111MiB/s (116MB/s), 111MiB/s-111MiB/s (116MB/s-116MB/s), io=774MiB (812MB), run=7001-7001msec

Disk stats (read/write):
  nvme0n1: ios=2250040/0, merge=0/0, ticks=11049/0, in_queue=11049, util=99.11%

$ rm /tmp/fio-output_512_2_libaio-bdev_aio_0.txt
rm: cannot remove '/tmp/fio-output_512_2_libaio-bdev_aio_0.txt': No such file or directory

$ /opt/fio/bin/fio --name="libaio-bdev_aio" --rw="randread" --size="12G" --iodepth="2" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_aio_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_2_libaio-bdev_aio_0.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_512_2_libaio-bdev_aio_0.txt
libaio-bdev_aio: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666789974,
  "timestamp_ms" : 1666789974420,
  "time" : "Wed Oct 26 15:12:54 2022",
  "jobs" : [
    {
      "jobname" : "libaio-bdev_aio",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-bdev_aio",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "2",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_aio_foo.conf",
        "filename" : "Nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 1065744896,
        "io_kbytes" : 1040766,
        "bw_bytes" : 152227524,
        "bw" : 148659,
        "iops" : 297319.240109,
        "runtime" : 7001,
        "total_ios" : 2081532,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1663,
          "max" : 22833,
          "mean" : 1889.934944,
          "stddev" : 127.676401,
          "N" : 2081532
        },
        "clat_ns" : {
          "min" : 241,
          "max" : 102011,
          "mean" : 4605.353980,
          "stddev" : 1741.728440,
          "N" : 2081533,
          "percentile" : {
            "1.000000" : 3952,
            "5.000000" : 4192,
            "10.000000" : 4256,
            "20.000000" : 4256,
            "30.000000" : 4320,
            "40.000000" : 4320,
            "50.000000" : 4384,
            "60.000000" : 4384,
            "70.000000" : 4384,
            "80.000000" : 4448,
            "90.000000" : 4512,
            "95.000000" : 4704,
            "99.000000" : 12608,
            "99.500000" : 17536,
            "99.900000" : 21888,
            "99.950000" : 22400,
            "99.990000" : 59648
          }
        },
        "lat_ns" : {
          "min" : 5129,
          "max" : 103885,
          "mean" : 6495.288902,
          "stddev" : 1745.563229,
          "N" : 2081533
        },
        "bw_min" : 147611,
        "bw_max" : 149598,
        "bw_agg" : 100.000000,
        "bw_mean" : 148687.142857,
        "bw_dev" : 617.469882,
        "bw_samples" : 14,
        "iops_min" : 295222,
        "iops_max" : 299196,
        "iops_mean" : 297374.357143,
        "iops_stddev" : 1234.727102,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 63.985714,
      "sys_cpu" : 35.985714,
      "ctx" : 15,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.100000,
        "2" : 99.989959,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.010000,
        "500" : 0.010000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 2.077076,
        "10" : 95.930641,
        "20" : 1.729688,
        "50" : 0.242177,
        "100" : 0.017968,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 2,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

libaio-bdev_aio: (groupid=0, jobs=1): err= 0: pid=7881: Wed Oct 26 15:12:54 2022
  read: IOPS=297k, BW=145MiB/s (152MB/s)(1016MiB/7001msec)
    slat (nsec): min=1663, max=22833, avg=1889.93, stdev=127.68
    clat (nsec): min=241, max=102011, avg=4605.35, stdev=1741.73
     lat (usec): min=5, max=103, avg= 6.50, stdev= 1.75
    clat percentiles (nsec):
     |  1.00th=[ 3952],  5.00th=[ 4192], 10.00th=[ 4256], 20.00th=[ 4256],
     | 30.00th=[ 4320], 40.00th=[ 4320], 50.00th=[ 4384], 60.00th=[ 4384],
     | 70.00th=[ 4384], 80.00th=[ 4448], 90.00th=[ 4512], 95.00th=[ 4704],
     | 99.00th=[12608], 99.50th=[17536], 99.90th=[21888], 99.95th=[22400],
     | 99.99th=[59648]
   bw (  KiB/s): min=147611, max=149598, per=100.00%, avg=148687.14, stdev=617.47, samples=14
   iops        : min=295222, max=299196, avg=297374.36, stdev=1234.73, samples=14
  lat (nsec)   : 250=0.01%, 500=0.01%
  lat (usec)   : 4=2.08%, 10=95.93%, 20=1.73%, 50=0.24%, 100=0.02%
  lat (usec)   : 250=0.01%
  cpu          : usr=63.99%, sys=35.99%, ctx=15, majf=0, minf=0
  IO depths    : 1=0.1%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=2081532,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=2

Run status group 0 (all jobs):
   READ: bw=145MiB/s (152MB/s), 145MiB/s-145MiB/s (152MB/s-152MB/s), io=1016MiB (1066MB), run=7001-7001msec

$ rm /tmp/fio-output_512_2_libaio-bdev_aio_1.txt
rm: cannot remove '/tmp/fio-output_512_2_libaio-bdev_aio_1.txt': No such file or directory

$ /opt/fio/bin/fio --name="libaio-bdev_aio" --rw="randread" --size="12G" --iodepth="2" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_aio_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_2_libaio-bdev_aio_1.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_512_2_libaio-bdev_aio_1.txt
libaio-bdev_aio: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666789989,
  "timestamp_ms" : 1666789989718,
  "time" : "Wed Oct 26 15:13:09 2022",
  "jobs" : [
    {
      "jobname" : "libaio-bdev_aio",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-bdev_aio",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "2",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_aio_foo.conf",
        "filename" : "Nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 1061161472,
        "io_kbytes" : 1036290,
        "bw_bytes" : 151572842,
        "bw" : 148020,
        "iops" : 296040.422797,
        "runtime" : 7001,
        "total_ios" : 2072579,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1683,
          "max" : 35486,
          "mean" : 1927.547332,
          "stddev" : 125.332926,
          "N" : 2072580
        },
        "clat_ns" : {
          "min" : 240,
          "max" : 98986,
          "mean" : 4604.571840,
          "stddev" : 1756.957642,
          "N" : 2072579,
          "percentile" : {
            "1.000000" : 3952,
            "5.000000" : 4192,
            "10.000000" : 4256,
            "20.000000" : 4256,
            "30.000000" : 4320,
            "40.000000" : 4320,
            "50.000000" : 4384,
            "60.000000" : 4384,
            "70.000000" : 4384,
            "80.000000" : 4448,
            "90.000000" : 4512,
            "95.000000" : 4704,
            "99.000000" : 12608,
            "99.500000" : 17536,
            "99.900000" : 21888,
            "99.950000" : 22400,
            "99.990000" : 61184
          }
        },
        "lat_ns" : {
          "min" : 5180,
          "max" : 100919,
          "mean" : 6532.119198,
          "stddev" : 1759.977965,
          "N" : 2072579
        },
        "bw_min" : 147349,
        "bw_max" : 148689,
        "bw_agg" : 100.000000,
        "bw_mean" : 148046.357143,
        "bw_dev" : 438.091070,
        "bw_samples" : 14,
        "iops_min" : 294698,
        "iops_max" : 297378,
        "iops_mean" : 296092.714286,
        "iops_stddev" : 876.182140,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 63.528571,
      "sys_cpu" : 36.442857,
      "ctx" : 11,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.100000,
        "2" : 99.989240,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.010000,
        "500" : 0.010000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 2.120884,
        "10" : 95.873450,
        "20" : 1.738317,
        "50" : 0.244237,
        "100" : 0.020506,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 2,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

libaio-bdev_aio: (groupid=0, jobs=1): err= 0: pid=7940: Wed Oct 26 15:13:09 2022
  read: IOPS=296k, BW=145MiB/s (152MB/s)(1012MiB/7001msec)
    slat (nsec): min=1683, max=35486, avg=1927.55, stdev=125.33
    clat (nsec): min=240, max=98986, avg=4604.57, stdev=1756.96
     lat (usec): min=5, max=100, avg= 6.53, stdev= 1.76
    clat percentiles (nsec):
     |  1.00th=[ 3952],  5.00th=[ 4192], 10.00th=[ 4256], 20.00th=[ 4256],
     | 30.00th=[ 4320], 40.00th=[ 4320], 50.00th=[ 4384], 60.00th=[ 4384],
     | 70.00th=[ 4384], 80.00th=[ 4448], 90.00th=[ 4512], 95.00th=[ 4704],
     | 99.00th=[12608], 99.50th=[17536], 99.90th=[21888], 99.95th=[22400],
     | 99.99th=[61184]
   bw (  KiB/s): min=147349, max=148689, per=100.00%, avg=148046.36, stdev=438.09, samples=14
   iops        : min=294698, max=297378, avg=296092.71, stdev=876.18, samples=14
  lat (nsec)   : 250=0.01%, 500=0.01%
  lat (usec)   : 4=2.12%, 10=95.87%, 20=1.74%, 50=0.24%, 100=0.02%
  cpu          : usr=63.53%, sys=36.44%, ctx=11, majf=0, minf=0
  IO depths    : 1=0.1%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=2072579,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=2

Run status group 0 (all jobs):
   READ: bw=145MiB/s (152MB/s), 145MiB/s-145MiB/s (152MB/s-152MB/s), io=1012MiB (1061MB), run=7001-7001msec

$ rm /tmp/fio-output_512_2_libaio-bdev_aio_2.txt
rm: cannot remove '/tmp/fio-output_512_2_libaio-bdev_aio_2.txt': No such file or directory

$ /opt/fio/bin/fio --name="libaio-bdev_aio" --rw="randread" --size="12G" --iodepth="2" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_aio_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_2_libaio-bdev_aio_2.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_512_1_io_uring_cmd-xnvme_2.txt
io_uring_cmd-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666789217,
  "timestamp_ms" : 1666789217545,
  "time" : "Wed Oct 26 15:00:17 2022",
  "jobs" : [
    {
      "jobname" : "io_uring_cmd-xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring_cmd-xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "1",
        "bs" : "512",
        "direct" : "0",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
        "xnvme_be" : "linux",
        "xnvme_async" : "io_uring_cmd",
        "xnvme_sync" : "nvme",
        "xnvme_admin" : "nvme",
        "xnvme_dev_nsid" : "1",
        "filename" : "/dev/ng0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 392771072,
        "io_kbytes" : 383565,
        "bw_bytes" : 56102138,
        "bw" : 54787,
        "iops" : 109574.489359,
        "runtime" : 7001,
        "total_ios" : 767131,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1423,
          "max" : 5861,
          "mean" : 1520.401970,
          "stddev" : 57.282899,
          "N" : 767131
        },
        "clat_ns" : {
          "min" : 1362,
          "max" : 75181,
          "mean" : 7308.434213,
          "stddev" : 994.829189,
          "N" : 767131,
          "percentile" : {
            "1.000000" : 6816,
            "5.000000" : 7008,
            "10.000000" : 7072,
            "20.000000" : 7072,
            "30.000000" : 7136,
            "40.000000" : 7200,
            "50.000000" : 7200,
            "60.000000" : 7264,
            "70.000000" : 7264,
            "80.000000" : 7328,
            "90.000000" : 7456,
            "95.000000" : 7520,
            "99.000000" : 10432,
            "99.500000" : 13632,
            "99.900000" : 15680,
            "99.950000" : 15808,
            "99.990000" : 59136
          }
        },
        "lat_ns" : {
          "min" : 6832,
          "max" : 76674,
          "mean" : 8828.836183,
          "stddev" : 997.829487,
          "N" : 767131
        },
        "bw_min" : 54620,
        "bw_max" : 55051,
        "bw_agg" : 100.000000,
        "bw_mean" : 54797.071429,
        "bw_dev" : 124.224031,
        "bw_samples" : 14,
        "iops_min" : 109240,
        "iops_max" : 110102,
        "iops_mean" : 109594.142857,
        "iops_stddev" : 248.448062,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 7.342857,
      "sys_cpu" : 37.014286,
      "ctx" : 767130,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 100.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.010000,
        "4" : 0.000000,
        "10" : 98.706740,
        "20" : 1.275532,
        "50" : 0.010000,
        "100" : 0.014861,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 1,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring_cmd-xnvme: (groupid=0, jobs=1): err= 0: pid=5169: Wed Oct 26 15:00:17 2022
  read: IOPS=110k, BW=53.5MiB/s (56.1MB/s)(375MiB/7001msec)
    slat (nsec): min=1423, max=5861, avg=1520.40, stdev=57.28
    clat (nsec): min=1362, max=75181, avg=7308.43, stdev=994.83
     lat (nsec): min=6832, max=76674, avg=8828.84, stdev=997.83
    clat percentiles (nsec):
     |  1.00th=[ 6816],  5.00th=[ 7008], 10.00th=[ 7072], 20.00th=[ 7072],
     | 30.00th=[ 7136], 40.00th=[ 7200], 50.00th=[ 7200], 60.00th=[ 7264],
     | 70.00th=[ 7264], 80.00th=[ 7328], 90.00th=[ 7456], 95.00th=[ 7520],
     | 99.00th=[10432], 99.50th=[13632], 99.90th=[15680], 99.95th=[15808],
     | 99.99th=[59136]
   bw (  KiB/s): min=54620, max=55051, per=100.00%, avg=54797.07, stdev=124.22, samples=14
   iops        : min=109240, max=110102, avg=109594.14, stdev=248.45, samples=14
  lat (usec)   : 2=0.01%, 10=98.71%, 20=1.28%, 50=0.01%, 100=0.01%
  cpu          : usr=7.34%, sys=37.01%, ctx=767130, majf=0, minf=0
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=767131,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=53.5MiB/s (56.1MB/s), 53.5MiB/s-53.5MiB/s (56.1MB/s-56.1MB/s), io=375MiB (393MB), run=7001-7001msec

$ cat /tmp/fio-output_512_2_libaio-bdev_aio_2.txt
libaio-bdev_aio: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666790005,
  "timestamp_ms" : 1666790005075,
  "time" : "Wed Oct 26 15:13:25 2022",
  "jobs" : [
    {
      "jobname" : "libaio-bdev_aio",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-bdev_aio",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "2",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_aio_foo.conf",
        "filename" : "Nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 1064973824,
        "io_kbytes" : 1040013,
        "bw_bytes" : 152117386,
        "bw" : 148552,
        "iops" : 297103.985145,
        "runtime" : 7001,
        "total_ios" : 2080025,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1663,
          "max" : 18003,
          "mean" : 1897.384686,
          "stddev" : 130.276176,
          "N" : 2080026
        },
        "clat_ns" : {
          "min" : 250,
          "max" : 117160,
          "mean" : 4608.007251,
          "stddev" : 1767.883717,
          "N" : 2080025,
          "percentile" : {
            "1.000000" : 3952,
            "5.000000" : 4192,
            "10.000000" : 4256,
            "20.000000" : 4256,
            "30.000000" : 4320,
            "40.000000" : 4320,
            "50.000000" : 4384,
            "60.000000" : 4384,
            "70.000000" : 4448,
            "80.000000" : 4448,
            "90.000000" : 4512,
            "95.000000" : 4704,
            "99.000000" : 12608,
            "99.500000" : 17536,
            "99.900000" : 21888,
            "99.950000" : 22400,
            "99.990000" : 61184
          }
        },
        "lat_ns" : {
          "min" : 5160,
          "max" : 119033,
          "mean" : 6505.391929,
          "stddev" : 1771.826567,
          "N" : 2080025
        },
        "bw_min" : 147892,
        "bw_max" : 149767,
        "bw_agg" : 100.000000,
        "bw_mean" : 148627.928571,
        "bw_dev" : 466.567740,
        "bw_samples" : 14,
        "iops_min" : 295785,
        "iops_max" : 299534,
        "iops_mean" : 297256.000000,
        "iops_stddev" : 932.970030,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 59.700000,
      "sys_cpu" : 40.271429,
      "ctx" : 10,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.100000,
        "2" : 99.989808,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.010000,
        "750" : 0.000000,
        "1000" : 0.010000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 2.073292,
        "10" : 95.884665,
        "20" : 1.778440,
        "50" : 0.241055,
        "100" : 0.020096,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 2,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

libaio-bdev_aio: (groupid=0, jobs=1): err= 0: pid=7997: Wed Oct 26 15:13:25 2022
  read: IOPS=297k, BW=145MiB/s (152MB/s)(1016MiB/7001msec)
    slat (nsec): min=1663, max=18003, avg=1897.38, stdev=130.28
    clat (nsec): min=250, max=117160, avg=4608.01, stdev=1767.88
     lat (usec): min=5, max=119, avg= 6.51, stdev= 1.77
    clat percentiles (nsec):
     |  1.00th=[ 3952],  5.00th=[ 4192], 10.00th=[ 4256], 20.00th=[ 4256],
     | 30.00th=[ 4320], 40.00th=[ 4320], 50.00th=[ 4384], 60.00th=[ 4384],
     | 70.00th=[ 4448], 80.00th=[ 4448], 90.00th=[ 4512], 95.00th=[ 4704],
     | 99.00th=[12608], 99.50th=[17536], 99.90th=[21888], 99.95th=[22400],
     | 99.99th=[61184]
   bw (  KiB/s): min=147892, max=149767, per=100.00%, avg=148627.93, stdev=466.57, samples=14
   iops        : min=295785, max=299534, avg=297256.00, stdev=932.97, samples=14
  lat (nsec)   : 500=0.01%, 1000=0.01%
  lat (usec)   : 4=2.07%, 10=95.88%, 20=1.78%, 50=0.24%, 100=0.02%
  lat (usec)   : 250=0.01%
  cpu          : usr=59.70%, sys=40.27%, ctx=10, majf=0, minf=0
  IO depths    : 1=0.1%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=2080025,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=2

Run status group 0 (all jobs):
   READ: bw=145MiB/s (152MB/s), 145MiB/s-145MiB/s (152MB/s-152MB/s), io=1016MiB (1065MB), run=7001-7001msec

$ rm /tmp/fio-output_512_2_libaio-bdev_xnvme_0.txt
rm: cannot remove '/tmp/fio-output_512_2_libaio-bdev_xnvme_0.txt': No such file or directory

$ /opt/fio/bin/fio --name="libaio-bdev_xnvme" --rw="randread" --size="12G" --iodepth="2" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_2_libaio-bdev_xnvme_0.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_512_2_libaio-bdev_xnvme_0.txt
libaio-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666790020,
  "timestamp_ms" : 1666790020382,
  "time" : "Wed Oct 26 15:13:40 2022",
  "jobs" : [
    {
      "jobname" : "libaio-bdev_xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-bdev_xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "2",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_xnvme_libaio.conf",
        "filename" : "Nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 1019141632,
        "io_kbytes" : 995255,
        "bw_bytes" : 145570865,
        "bw" : 142159,
        "iops" : 284317.954578,
        "runtime" : 7001,
        "total_ios" : 1990510,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1693,
          "max" : 23505,
          "mean" : 1928.243363,
          "stddev" : 145.937912,
          "N" : 1990510
        },
        "clat_ns" : {
          "min" : 611,
          "max" : 114164,
          "mean" : 4850.434752,
          "stddev" : 1749.968240,
          "N" : 1990511,
          "percentile" : {
            "1.000000" : 4256,
            "5.000000" : 4384,
            "10.000000" : 4384,
            "20.000000" : 4512,
            "30.000000" : 4576,
            "40.000000" : 4576,
            "50.000000" : 4640,
            "60.000000" : 4640,
            "70.000000" : 4640,
            "80.000000" : 4704,
            "90.000000" : 4768,
            "95.000000" : 4896,
            "99.000000" : 12736,
            "99.500000" : 18048,
            "99.900000" : 21888,
            "99.950000" : 22144,
            "99.990000" : 60672
          }
        },
        "lat_ns" : {
          "min" : 5460,
          "max" : 116057,
          "mean" : 6778.678102,
          "stddev" : 1753.594753,
          "N" : 1990511
        },
        "bw_min" : 141762,
        "bw_max" : 142974,
        "bw_agg" : 100.000000,
        "bw_mean" : 142184.785714,
        "bw_dev" : 341.212809,
        "bw_samples" : 14,
        "iops_min" : 283524,
        "iops_max" : 285949,
        "iops_mean" : 284369.642857,
        "iops_stddev" : 682.603569,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 56.500000,
      "sys_cpu" : 43.471429,
      "ctx" : 7,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.100000,
        "2" : 99.977795,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.010000,
        "1000" : 0.010000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.017684,
        "10" : 97.705714,
        "20" : 1.999236,
        "50" : 0.251091,
        "100" : 0.019744,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 2,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

libaio-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=8054: Wed Oct 26 15:13:40 2022
  read: IOPS=284k, BW=139MiB/s (146MB/s)(972MiB/7001msec)
    slat (nsec): min=1693, max=23505, avg=1928.24, stdev=145.94
    clat (nsec): min=611, max=114164, avg=4850.43, stdev=1749.97
     lat (usec): min=5, max=116, avg= 6.78, stdev= 1.75
    clat percentiles (nsec):
     |  1.00th=[ 4256],  5.00th=[ 4384], 10.00th=[ 4384], 20.00th=[ 4512],
     | 30.00th=[ 4576], 40.00th=[ 4576], 50.00th=[ 4640], 60.00th=[ 4640],
     | 70.00th=[ 4640], 80.00th=[ 4704], 90.00th=[ 4768], 95.00th=[ 4896],
     | 99.00th=[12736], 99.50th=[18048], 99.90th=[21888], 99.95th=[22144],
     | 99.99th=[60672]
   bw (  KiB/s): min=141762, max=142974, per=100.00%, avg=142184.79, stdev=341.21, samples=14
   iops        : min=283524, max=285949, avg=284369.64, stdev=682.60, samples=14
  lat (nsec)   : 750=0.01%, 1000=0.01%
  lat (usec)   : 4=0.02%, 10=97.71%, 20=2.00%, 50=0.25%, 100=0.02%
  lat (usec)   : 250=0.01%
  cpu          : usr=56.50%, sys=43.47%, ctx=7, majf=0, minf=0
  IO depths    : 1=0.1%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1990510,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=2

Run status group 0 (all jobs):
   READ: bw=139MiB/s (146MB/s), 139MiB/s-139MiB/s (146MB/s-146MB/s), io=972MiB (1019MB), run=7001-7001msec

$ rm /tmp/fio-output_512_2_libaio-bdev_xnvme_1.txt
rm: cannot remove '/tmp/fio-output_512_2_libaio-bdev_xnvme_1.txt': No such file or directory

$ /opt/fio/bin/fio --name="libaio-bdev_xnvme" --rw="randread" --size="12G" --iodepth="2" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_2_libaio-bdev_xnvme_1.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_512_2_libaio-bdev_xnvme_1.txt
libaio-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666790035,
  "timestamp_ms" : 1666790035564,
  "time" : "Wed Oct 26 15:13:55 2022",
  "jobs" : [
    {
      "jobname" : "libaio-bdev_xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-bdev_xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "2",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_xnvme_libaio.conf",
        "filename" : "Nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 1025831424,
        "io_kbytes" : 1001788,
        "bw_bytes" : 146526413,
        "bw" : 143092,
        "iops" : 286184.259392,
        "runtime" : 7001,
        "total_ios" : 2003576,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1713,
          "max" : 22923,
          "mean" : 1939.483181,
          "stddev" : 128.999116,
          "N" : 2003576
        },
        "clat_ns" : {
          "min" : 652,
          "max" : 114214,
          "mean" : 4820.917562,
          "stddev" : 1748.341345,
          "N" : 2003577,
          "percentile" : {
            "1.000000" : 4256,
            "5.000000" : 4384,
            "10.000000" : 4448,
            "20.000000" : 4512,
            "30.000000" : 4512,
            "40.000000" : 4576,
            "50.000000" : 4576,
            "60.000000" : 4576,
            "70.000000" : 4640,
            "80.000000" : 4640,
            "90.000000" : 4768,
            "95.000000" : 4896,
            "99.000000" : 12736,
            "99.500000" : 18048,
            "99.900000" : 21888,
            "99.950000" : 22144,
            "99.990000" : 61184
          }
        },
        "lat_ns" : {
          "min" : 5680,
          "max" : 116118,
          "mean" : 6760.400710,
          "stddev" : 1752.206198,
          "N" : 2003577
        },
        "bw_min" : 142671,
        "bw_max" : 143940,
        "bw_agg" : 100.000000,
        "bw_mean" : 143196.692308,
        "bw_dev" : 456.645629,
        "bw_samples" : 13,
        "iops_min" : 285342,
        "iops_max" : 287880,
        "iops_mean" : 286393.538462,
        "iops_stddev" : 913.368638,
        "iops_samples" : 13
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 55.757143,
      "sys_cpu" : 44.228571,
      "ctx" : 9,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.100000,
        "2" : 99.987323,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.010000,
        "1000" : 0.010000
      },
      "latency_us" : {
        "2" : 0.010000,
        "4" : 0.023558,
        "10" : 97.738543,
        "20" : 1.962641,
        "50" : 0.251900,
        "100" : 0.019815,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 2,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

libaio-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=8112: Wed Oct 26 15:13:55 2022
  read: IOPS=286k, BW=140MiB/s (147MB/s)(978MiB/7001msec)
    slat (nsec): min=1713, max=22923, avg=1939.48, stdev=129.00
    clat (nsec): min=652, max=114214, avg=4820.92, stdev=1748.34
     lat (usec): min=5, max=116, avg= 6.76, stdev= 1.75
    clat percentiles (nsec):
     |  1.00th=[ 4256],  5.00th=[ 4384], 10.00th=[ 4448], 20.00th=[ 4512],
     | 30.00th=[ 4512], 40.00th=[ 4576], 50.00th=[ 4576], 60.00th=[ 4576],
     | 70.00th=[ 4640], 80.00th=[ 4640], 90.00th=[ 4768], 95.00th=[ 4896],
     | 99.00th=[12736], 99.50th=[18048], 99.90th=[21888], 99.95th=[22144],
     | 99.99th=[61184]
   bw (  KiB/s): min=142671, max=143940, per=100.00%, avg=143196.69, stdev=456.65, samples=13
   iops        : min=285342, max=287880, avg=286393.54, stdev=913.37, samples=13
  lat (nsec)   : 750=0.01%, 1000=0.01%
  lat (usec)   : 2=0.01%, 4=0.02%, 10=97.74%, 20=1.96%, 50=0.25%
  lat (usec)   : 100=0.02%, 250=0.01%
  cpu          : usr=55.76%, sys=44.23%, ctx=9, majf=0, minf=0
  IO depths    : 1=0.1%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=2003576,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=2

Run status group 0 (all jobs):
   READ: bw=140MiB/s (147MB/s), 140MiB/s-140MiB/s (147MB/s-147MB/s), io=978MiB (1026MB), run=7001-7001msec

$ rm /tmp/fio-output_512_2_libaio-bdev_xnvme_2.txt
rm: cannot remove '/tmp/fio-output_512_2_libaio-bdev_xnvme_2.txt': No such file or directory

$ /opt/fio/bin/fio --name="libaio-bdev_xnvme" --rw="randread" --size="12G" --iodepth="2" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_2_libaio-bdev_xnvme_2.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_512_2_libaio-bdev_xnvme_2.txt
libaio-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666790050,
  "timestamp_ms" : 1666790050783,
  "time" : "Wed Oct 26 15:14:10 2022",
  "jobs" : [
    {
      "jobname" : "libaio-bdev_xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-bdev_xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "2",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_xnvme_libaio.conf",
        "filename" : "Nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 1025968128,
        "io_kbytes" : 1001922,
        "bw_bytes" : 146566875,
        "bw" : 143131,
        "iops" : 286263.142857,
        "runtime" : 7000,
        "total_ios" : 2003842,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1683,
          "max" : 22953,
          "mean" : 1917.783290,
          "stddev" : 143.998960,
          "N" : 2003843
        },
        "clat_ns" : {
          "min" : 651,
          "max" : 119825,
          "mean" : 4827.615465,
          "stddev" : 1728.532459,
          "N" : 2003842,
          "percentile" : {
            "1.000000" : 4256,
            "5.000000" : 4384,
            "10.000000" : 4384,
            "20.000000" : 4448,
            "30.000000" : 4512,
            "40.000000" : 4576,
            "50.000000" : 4640,
            "60.000000" : 4640,
            "70.000000" : 4640,
            "80.000000" : 4704,
            "90.000000" : 4768,
            "95.000000" : 4960,
            "99.000000" : 12736,
            "99.500000" : 18048,
            "99.900000" : 21888,
            "99.950000" : 22144,
            "99.990000" : 59648
          }
        },
        "lat_ns" : {
          "min" : 5410,
          "max" : 121838,
          "mean" : 6745.398757,
          "stddev" : 1732.651265,
          "N" : 2003842
        },
        "bw_min" : 142876,
        "bw_max" : 143492,
        "bw_agg" : 100.000000,
        "bw_mean" : 143137.428571,
        "bw_dev" : 166.771476,
        "bw_samples" : 14,
        "iops_min" : 285752,
        "iops_max" : 286984,
        "iops_mean" : 286274.928571,
        "iops_stddev" : 333.438142,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 55.185714,
      "sys_cpu" : 44.785714,
      "ctx" : 7,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.100000,
        "2" : 99.978691,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.010000,
        "1000" : 0.010000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.016418,
        "10" : 97.727366,
        "20" : 1.985785,
        "50" : 0.247275,
        "100" : 0.017267,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 2,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

libaio-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=8172: Wed Oct 26 15:14:10 2022
  read: IOPS=286k, BW=140MiB/s (147MB/s)(978MiB/7000msec)
    slat (nsec): min=1683, max=22953, avg=1917.78, stdev=144.00
    clat (nsec): min=651, max=119825, avg=4827.62, stdev=1728.53
     lat (usec): min=5, max=121, avg= 6.75, stdev= 1.73
    clat percentiles (nsec):
     |  1.00th=[ 4256],  5.00th=[ 4384], 10.00th=[ 4384], 20.00th=[ 4448],
     | 30.00th=[ 4512], 40.00th=[ 4576], 50.00th=[ 4640], 60.00th=[ 4640],
     | 70.00th=[ 4640], 80.00th=[ 4704], 90.00th=[ 4768], 95.00th=[ 4960],
     | 99.00th=[12736], 99.50th=[18048], 99.90th=[21888], 99.95th=[22144],
     | 99.99th=[59648]
   bw (  KiB/s): min=142876, max=143492, per=100.00%, avg=143137.43, stdev=166.77, samples=14
   iops        : min=285752, max=286984, avg=286274.93, stdev=333.44, samples=14
  lat (nsec)   : 750=0.01%, 1000=0.01%
  lat (usec)   : 4=0.02%, 10=97.73%, 20=1.99%, 50=0.25%, 100=0.02%
  lat (usec)   : 250=0.01%
  cpu          : usr=55.19%, sys=44.79%, ctx=7, majf=0, minf=0
  IO depths    : 1=0.1%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=2003842,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=2

Run status group 0 (all jobs):
   READ: bw=140MiB/s (147MB/s), 140MiB/s-140MiB/s (147MB/s-147MB/s), io=978MiB (1026MB), run=7000-7000msec

$ rm /tmp/fio-output_512_1_io_uring_cmd-bdev_xnvme_0.txt
rm: cannot remove '/tmp/fio-output_512_1_io_uring_cmd-bdev_xnvme_0.txt': No such file or directory

$ rm /tmp/fio-output_512_2_libaio-xnvme_0.txt
rm: cannot remove '/tmp/fio-output_512_2_libaio-xnvme_0.txt': No such file or directory

$ /opt/fio/bin/fio --name="libaio-xnvme" --rw="randread" --size="12G" --iodepth="2" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="libaio" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_2_libaio-xnvme_0.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_2_libaio-xnvme_0.txt
libaio-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666790064,
  "timestamp_ms" : 1666790064713,
  "time" : "Wed Oct 26 15:14:24 2022",
  "jobs" : [
    {
      "jobname" : "libaio-xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "2",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
        "xnvme_be" : "linux",
        "xnvme_async" : "libaio",
        "xnvme_sync" : "nvme",
        "xnvme_admin" : "nvme",
        "xnvme_dev_nsid" : "1",
        "filename" : "/dev/nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 840821760,
        "io_kbytes" : 821115,
        "bw_bytes" : 120100237,
        "bw" : 117285,
        "iops" : 234570.489930,
        "runtime" : 7001,
        "total_ios" : 1642228,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1503,
          "max" : 27501,
          "mean" : 1736.627195,
          "stddev" : 156.192241,
          "N" : 1642229
        },
        "clat_ns" : {
          "min" : 2836,
          "max" : 121638,
          "mean" : 6548.460939,
          "stddev" : 1775.773390,
          "N" : 1642229,
          "percentile" : {
            "1.000000" : 5920,
            "5.000000" : 5920,
            "10.000000" : 5984,
            "20.000000" : 6048,
            "30.000000" : 6112,
            "40.000000" : 6176,
            "50.000000" : 6240,
            "60.000000" : 6304,
            "70.000000" : 6368,
            "80.000000" : 6432,
            "90.000000" : 7136,
            "95.000000" : 7584,
            "99.000000" : 15040,
            "99.500000" : 18560,
            "99.900000" : 25984,
            "99.950000" : 26496,
            "99.990000" : 62208
          }
        },
        "lat_ns" : {
          "min" : 6603,
          "max" : 123481,
          "mean" : 8285.088133,
          "stddev" : 1774.881999,
          "N" : 1642229
        },
        "bw_min" : 116619,
        "bw_max" : 117728,
        "bw_agg" : 100.000000,
        "bw_mean" : 117305.857143,
        "bw_dev" : 299.454853,
        "bw_samples" : 14,
        "iops_min" : 233238,
        "iops_max" : 235456,
        "iops_mean" : 234611.928571,
        "iops_stddev" : 598.818126,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 19.800000,
      "sys_cpu" : 57.285714,
      "ctx" : 824948,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 100.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010000,
        "10" : 97.830691,
        "20" : 1.792078,
        "50" : 0.349099,
        "100" : 0.020764,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 2,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

libaio-xnvme: (groupid=0, jobs=1): err= 0: pid=8219: Wed Oct 26 15:14:24 2022
  read: IOPS=235k, BW=115MiB/s (120MB/s)(802MiB/7001msec)
    slat (nsec): min=1503, max=27501, avg=1736.63, stdev=156.19
    clat (usec): min=2, max=121, avg= 6.55, stdev= 1.78
     lat (usec): min=6, max=123, avg= 8.29, stdev= 1.77
    clat percentiles (nsec):
     |  1.00th=[ 5920],  5.00th=[ 5920], 10.00th=[ 5984], 20.00th=[ 6048],
     | 30.00th=[ 6112], 40.00th=[ 6176], 50.00th=[ 6240], 60.00th=[ 6304],
     | 70.00th=[ 6368], 80.00th=[ 6432], 90.00th=[ 7136], 95.00th=[ 7584],
     | 99.00th=[15040], 99.50th=[18560], 99.90th=[25984], 99.95th=[26496],
     | 99.99th=[62208]
   bw (  KiB/s): min=116619, max=117728, per=100.00%, avg=117305.86, stdev=299.45, samples=14
   iops        : min=233238, max=235456, avg=234611.93, stdev=598.82, samples=14
  lat (usec)   : 4=0.01%, 10=97.83%, 20=1.79%, 50=0.35%, 100=0.02%
  lat (usec)   : 250=0.01%
  cpu          : usr=19.80%, sys=57.29%, ctx=824948, majf=0, minf=0
  IO depths    : 1=0.0%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1642228,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=2

Run status group 0 (all jobs):
   READ: bw=115MiB/s (120MB/s), 115MiB/s-115MiB/s (120MB/s-120MB/s), io=802MiB (841MB), run=7001-7001msec

$ rm /tmp/fio-output_512_2_libaio-xnvme_1.txt
rm: cannot remove '/tmp/fio-output_512_2_libaio-xnvme_1.txt': No such file or directory

$ /opt/fio/bin/fio --name="libaio-xnvme" --rw="randread" --size="12G" --iodepth="2" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="libaio" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_2_libaio-xnvme_1.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_2_libaio-xnvme_1.txt
libaio-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666790078,
  "timestamp_ms" : 1666790078614,
  "time" : "Wed Oct 26 15:14:38 2022",
  "jobs" : [
    {
      "jobname" : "libaio-xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "2",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
        "xnvme_be" : "linux",
        "xnvme_async" : "libaio",
        "xnvme_sync" : "nvme",
        "xnvme_admin" : "nvme",
        "xnvme_dev_nsid" : "1",
        "filename" : "/dev/nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 837950976,
        "io_kbytes" : 818311,
        "bw_bytes" : 119690183,
        "bw" : 116884,
        "iops" : 233769.747179,
        "runtime" : 7001,
        "total_ios" : 1636622,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1513,
          "max" : 34655,
          "mean" : 1732.315724,
          "stddev" : 166.086574,
          "N" : 1636622
        },
        "clat_ns" : {
          "min" : 2925,
          "max" : 119975,
          "mean" : 6582.195259,
          "stddev" : 1784.605938,
          "N" : 1636623,
          "percentile" : {
            "1.000000" : 5920,
            "5.000000" : 5984,
            "10.000000" : 5984,
            "20.000000" : 6048,
            "30.000000" : 6112,
            "40.000000" : 6176,
            "50.000000" : 6304,
            "60.000000" : 6368,
            "70.000000" : 6432,
            "80.000000" : 6496,
            "90.000000" : 7072,
            "95.000000" : 7584,
            "99.000000" : 15296,
            "99.500000" : 18560,
            "99.900000" : 25984,
            "99.950000" : 26496,
            "99.990000" : 62720
          }
        },
        "lat_ns" : {
          "min" : 5931,
          "max" : 121779,
          "mean" : 8314.510898,
          "stddev" : 1783.242623,
          "N" : 1636623
        },
        "bw_min" : 116388,
        "bw_max" : 117454,
        "bw_agg" : 100.000000,
        "bw_mean" : 116959.071429,
        "bw_dev" : 338.761878,
        "bw_samples" : 14,
        "iops_min" : 232776,
        "iops_max" : 234908,
        "iops_mean" : 233918.357143,
        "iops_stddev" : 677.455262,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 20.128571,
      "sys_cpu" : 56.842857,
      "ctx" : 822028,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 100.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010000,
        "10" : 97.817945,
        "20" : 1.790823,
        "50" : 0.360560,
        "100" : 0.022241,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 2,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

libaio-xnvme: (groupid=0, jobs=1): err= 0: pid=8268: Wed Oct 26 15:14:38 2022
  read: IOPS=234k, BW=114MiB/s (120MB/s)(799MiB/7001msec)
    slat (nsec): min=1513, max=34655, avg=1732.32, stdev=166.09
    clat (usec): min=2, max=119, avg= 6.58, stdev= 1.78
     lat (usec): min=5, max=121, avg= 8.31, stdev= 1.78
    clat percentiles (nsec):
     |  1.00th=[ 5920],  5.00th=[ 5984], 10.00th=[ 5984], 20.00th=[ 6048],
     | 30.00th=[ 6112], 40.00th=[ 6176], 50.00th=[ 6304], 60.00th=[ 6368],
     | 70.00th=[ 6432], 80.00th=[ 6496], 90.00th=[ 7072], 95.00th=[ 7584],
     | 99.00th=[15296], 99.50th=[18560], 99.90th=[25984], 99.95th=[26496],
     | 99.99th=[62720]
   bw (  KiB/s): min=116388, max=117454, per=100.00%, avg=116959.07, stdev=338.76, samples=14
   iops        : min=232776, max=234908, avg=233918.36, stdev=677.46, samples=14
  lat (usec)   : 4=0.01%, 10=97.82%, 20=1.79%, 50=0.36%, 100=0.02%
  lat (usec)   : 250=0.01%
  cpu          : usr=20.13%, sys=56.84%, ctx=822028, majf=0, minf=0
  IO depths    : 1=0.0%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1636622,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=2

Run status group 0 (all jobs):
   READ: bw=114MiB/s (120MB/s), 114MiB/s-114MiB/s (120MB/s-120MB/s), io=799MiB (838MB), run=7001-7001msec

$ rm /tmp/fio-output_512_2_libaio-xnvme_2.txt
rm: cannot remove '/tmp/fio-output_512_2_libaio-xnvme_2.txt': No such file or directory

$ /opt/fio/bin/fio --name="libaio-xnvme" --rw="randread" --size="12G" --iodepth="2" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="libaio" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_2_libaio-xnvme_2.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_2_libaio-xnvme_2.txt
libaio-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666790092,
  "timestamp_ms" : 1666790092370,
  "time" : "Wed Oct 26 15:14:52 2022",
  "jobs" : [
    {
      "jobname" : "libaio-xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "2",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
        "xnvme_be" : "linux",
        "xnvme_async" : "libaio",
        "xnvme_sync" : "nvme",
        "xnvme_admin" : "nvme",
        "xnvme_dev_nsid" : "1",
        "filename" : "/dev/nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 836393472,
        "io_kbytes" : 816790,
        "bw_bytes" : 119467714,
        "bw" : 116667,
        "iops" : 233335.237823,
        "runtime" : 7001,
        "total_ios" : 1633580,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1512,
          "max" : 53531,
          "mean" : 1744.740057,
          "stddev" : 169.297769,
          "N" : 1633580
        },
        "clat_ns" : {
          "min" : 2875,
          "max" : 119985,
          "mean" : 6578.702695,
          "stddev" : 1794.962820,
          "N" : 1633581,
          "percentile" : {
            "1.000000" : 5920,
            "5.000000" : 5984,
            "10.000000" : 5984,
            "20.000000" : 6048,
            "30.000000" : 6112,
            "40.000000" : 6176,
            "50.000000" : 6304,
            "60.000000" : 6304,
            "70.000000" : 6368,
            "80.000000" : 6496,
            "90.000000" : 7136,
            "95.000000" : 7584,
            "99.000000" : 15296,
            "99.500000" : 18560,
            "99.900000" : 25984,
            "99.950000" : 26496,
            "99.990000" : 62720
          }
        },
        "lat_ns" : {
          "min" : 6793,
          "max" : 121838,
          "mean" : 8323.442636,
          "stddev" : 1795.815444,
          "N" : 1633581
        },
        "bw_min" : 115883,
        "bw_max" : 117515,
        "bw_agg" : 100.000000,
        "bw_mean" : 116731.142857,
        "bw_dev" : 532.190951,
        "bw_samples" : 14,
        "iops_min" : 231766,
        "iops_max" : 235030,
        "iops_mean" : 233462.428571,
        "iops_stddev" : 1064.482230,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 20.028571,
      "sys_cpu" : 57.300000,
      "ctx" : 820575,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 100.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010000,
        "10" : 97.820186,
        "20" : 1.795872,
        "50" : 0.354130,
        "100" : 0.022833,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 2,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

libaio-xnvme: (groupid=0, jobs=1): err= 0: pid=8316: Wed Oct 26 15:14:52 2022
  read: IOPS=233k, BW=114MiB/s (119MB/s)(798MiB/7001msec)
    slat (nsec): min=1512, max=53531, avg=1744.74, stdev=169.30
    clat (usec): min=2, max=119, avg= 6.58, stdev= 1.79
     lat (usec): min=6, max=121, avg= 8.32, stdev= 1.80
    clat percentiles (nsec):
     |  1.00th=[ 5920],  5.00th=[ 5984], 10.00th=[ 5984], 20.00th=[ 6048],
     | 30.00th=[ 6112], 40.00th=[ 6176], 50.00th=[ 6304], 60.00th=[ 6304],
     | 70.00th=[ 6368], 80.00th=[ 6496], 90.00th=[ 7136], 95.00th=[ 7584],
     | 99.00th=[15296], 99.50th=[18560], 99.90th=[25984], 99.95th=[26496],
     | 99.99th=[62720]
   bw (  KiB/s): min=115883, max=117515, per=100.00%, avg=116731.14, stdev=532.19, samples=14
   iops        : min=231766, max=235030, avg=233462.43, stdev=1064.48, samples=14
  lat (usec)   : 4=0.01%, 10=97.82%, 20=1.80%, 50=0.35%, 100=0.02%
  lat (usec)   : 250=0.01%
  cpu          : usr=20.03%, sys=57.30%, ctx=820575, majf=0, minf=0
  IO depths    : 1=0.0%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1633580,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=2

Run status group 0 (all jobs):
   READ: bw=114MiB/s (119MB/s), 114MiB/s-114MiB/s (119MB/s-119MB/s), io=798MiB (836MB), run=7001-7001msec

$ rm /tmp/fio-output_512_4_io_uring_cmd-reference_0.txt
rm: cannot remove '/tmp/fio-output_512_4_io_uring_cmd-reference_0.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring_cmd-bdev_xnvme" --rw="randread" --size="12G" --iodepth="1" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_1_io_uring_cmd-bdev_xnvme_0.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created

$ /opt/fio/bin/fio --name="io_uring_cmd-reference" --rw="randread" --size="12G" --iodepth="4" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring_cmd" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_512_4_io_uring_cmd-reference_0.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_4_io_uring_cmd-reference_0.txt
io_uring_cmd-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=io_uring_cmd, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666790106,
  "timestamp_ms" : 1666790106140,
  "time" : "Wed Oct 26 15:15:06 2022",
  "jobs" : [
    {
      "jobname" : "io_uring_cmd-reference",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring_cmd-reference",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "4",
        "bs" : "512",
        "direct" : "0",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "io_uring_cmd",
        "filename" : "/dev/ng0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 1580787200,
        "io_kbytes" : 1543737,
        "bw_bytes" : 225794486,
        "bw" : 220502,
        "iops" : 441004.285102,
        "runtime" : 7001,
        "total_ios" : 3087471,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1172,
          "max" : 41017,
          "mean" : 1789.167935,
          "stddev" : 375.554215,
          "N" : 3087472
        },
        "clat_ns" : {
          "min" : 100,
          "max" : 122480,
          "mean" : 7011.723363,
          "stddev" : 2863.847161,
          "N" : 3087474,
          "percentile" : {
            "1.000000" : 4128,
            "5.000000" : 4704,
            "10.000000" : 4768,
            "20.000000" : 6112,
            "30.000000" : 6304,
            "40.000000" : 6368,
            "50.000000" : 6432,
            "60.000000" : 6624,
            "70.000000" : 6944,
            "80.000000" : 7584,
            "90.000000" : 8512,
            "95.000000" : 9280,
            "99.000000" : 20864,
            "99.500000" : 25216,
            "99.900000" : 27008,
            "99.950000" : 33536,
            "99.990000" : 68096
          }
        },
        "lat_ns" : {
          "min" : 5751,
          "max" : 123712,
          "mean" : 8800.891246,
          "stddev" : 2837.715863,
          "N" : 3087474
        },
        "bw_min" : 219155,
        "bw_max" : 221950,
        "bw_agg" : 100.000000,
        "bw_mean" : 220618.285714,
        "bw_dev" : 970.234264,
        "bw_samples" : 14,
        "iops_min" : 438310,
        "iops_max" : 443900,
        "iops_mean" : 441236.785714,
        "iops_stddev" : 1940.453523,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 15.271429,
      "sys_cpu" : 80.428571,
      "ctx" : 35740,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.010000,
        "4" : 0.016356,
        "10" : 95.917889,
        "20" : 2.579652,
        "50" : 1.446815,
        "100" : 0.039093,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 4,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring_cmd-reference: (groupid=0, jobs=1): err= 0: pid=8364: Wed Oct 26 15:15:06 2022
  read: IOPS=441k, BW=215MiB/s (226MB/s)(1508MiB/7001msec)
    slat (nsec): min=1172, max=41017, avg=1789.17, stdev=375.55
    clat (nsec): min=100, max=122480, avg=7011.72, stdev=2863.85
     lat (usec): min=5, max=123, avg= 8.80, stdev= 2.84
    clat percentiles (nsec):
     |  1.00th=[ 4128],  5.00th=[ 4704], 10.00th=[ 4768], 20.00th=[ 6112],
     | 30.00th=[ 6304], 40.00th=[ 6368], 50.00th=[ 6432], 60.00th=[ 6624],
     | 70.00th=[ 6944], 80.00th=[ 7584], 90.00th=[ 8512], 95.00th=[ 9280],
     | 99.00th=[20864], 99.50th=[25216], 99.90th=[27008], 99.95th=[33536],
     | 99.99th=[68096]
   bw (  KiB/s): min=219155, max=221950, per=100.00%, avg=220618.29, stdev=970.23, samples=14
   iops        : min=438310, max=443900, avg=441236.79, stdev=1940.45, samples=14
  lat (nsec)   : 250=0.01%
  lat (usec)   : 2=0.01%, 4=0.02%, 10=95.92%, 20=2.58%, 50=1.45%
  lat (usec)   : 100=0.04%, 250=0.01%
  cpu          : usr=15.27%, sys=80.43%, ctx=35740, majf=0, minf=0
  IO depths    : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=3087471,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: bw=215MiB/s (226MB/s), 215MiB/s-215MiB/s (226MB/s-226MB/s), io=1508MiB (1581MB), run=7001-7001msec

$ rm /tmp/fio-output_512_4_io_uring_cmd-reference_1.txt
rm: cannot remove '/tmp/fio-output_512_4_io_uring_cmd-reference_1.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring_cmd-reference" --rw="randread" --size="12G" --iodepth="4" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring_cmd" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_512_4_io_uring_cmd-reference_1.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_4_io_uring_cmd-reference_1.txt
io_uring_cmd-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=io_uring_cmd, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666790119,
  "timestamp_ms" : 1666790119919,
  "time" : "Wed Oct 26 15:15:19 2022",
  "jobs" : [
    {
      "jobname" : "io_uring_cmd-reference",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring_cmd-reference",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "4",
        "bs" : "512",
        "direct" : "0",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "io_uring_cmd",
        "filename" : "/dev/ng0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 1603376128,
        "io_kbytes" : 1565797,
        "bw_bytes" : 229021015,
        "bw" : 223653,
        "iops" : 447306.099129,
        "runtime" : 7001,
        "total_ios" : 3131590,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1162,
          "max" : 24146,
          "mean" : 1765.691817,
          "stddev" : 356.119959,
          "N" : 3131591
        },
        "clat_ns" : {
          "min" : 90,
          "max" : 125896,
          "mean" : 6918.897432,
          "stddev" : 2872.697041,
          "N" : 3131593,
          "percentile" : {
            "1.000000" : 4080,
            "5.000000" : 4640,
            "10.000000" : 4704,
            "20.000000" : 6048,
            "30.000000" : 6176,
            "40.000000" : 6240,
            "50.000000" : 6304,
            "60.000000" : 6432,
            "70.000000" : 6816,
            "80.000000" : 7456,
            "90.000000" : 8512,
            "95.000000" : 9152,
            "99.000000" : 20864,
            "99.500000" : 25216,
            "99.900000" : 27008,
            "99.950000" : 33024,
            "99.990000" : 68096
          }
        },
        "lat_ns" : {
          "min" : 5511,
          "max" : 127119,
          "mean" : 8684.589193,
          "stddev" : 2830.926192,
          "N" : 3131593
        },
        "bw_min" : 222821,
        "bw_max" : 224344,
        "bw_agg" : 100.000000,
        "bw_mean" : 223701.928571,
        "bw_dev" : 419.962354,
        "bw_samples" : 14,
        "iops_min" : 445642,
        "iops_max" : 448688,
        "iops_mean" : 447403.571429,
        "iops_stddev" : 839.997868,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 17.257143,
      "sys_cpu" : 78.285714,
      "ctx" : 39299,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.010000,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.010000,
        "4" : 0.170808,
        "10" : 95.608844,
        "20" : 2.736278,
        "50" : 1.445496,
        "100" : 0.037393,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 4,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring_cmd-reference: (groupid=0, jobs=1): err= 0: pid=8410: Wed Oct 26 15:15:19 2022
  read: IOPS=447k, BW=218MiB/s (229MB/s)(1529MiB/7001msec)
    slat (nsec): min=1162, max=24146, avg=1765.69, stdev=356.12
    clat (nsec): min=90, max=125896, avg=6918.90, stdev=2872.70
     lat (usec): min=5, max=127, avg= 8.68, stdev= 2.83
    clat percentiles (nsec):
     |  1.00th=[ 4080],  5.00th=[ 4640], 10.00th=[ 4704], 20.00th=[ 6048],
     | 30.00th=[ 6176], 40.00th=[ 6240], 50.00th=[ 6304], 60.00th=[ 6432],
     | 70.00th=[ 6816], 80.00th=[ 7456], 90.00th=[ 8512], 95.00th=[ 9152],
     | 99.00th=[20864], 99.50th=[25216], 99.90th=[27008], 99.95th=[33024],
     | 99.99th=[68096]
   bw (  KiB/s): min=222821, max=224344, per=100.00%, avg=223701.93, stdev=419.96, samples=14
   iops        : min=445642, max=448688, avg=447403.57, stdev=840.00, samples=14
  lat (nsec)   : 100=0.01%, 250=0.01%
  lat (usec)   : 2=0.01%, 4=0.17%, 10=95.61%, 20=2.74%, 50=1.45%
  lat (usec)   : 100=0.04%, 250=0.01%
  cpu          : usr=17.26%, sys=78.29%, ctx=39299, majf=0, minf=0
  IO depths    : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=3131590,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: bw=218MiB/s (229MB/s), 218MiB/s-218MiB/s (229MB/s-229MB/s), io=1529MiB (1603MB), run=7001-7001msec

$ rm /tmp/fio-output_512_4_io_uring_cmd-reference_2.txt
rm: cannot remove '/tmp/fio-output_512_4_io_uring_cmd-reference_2.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring_cmd-reference" --rw="randread" --size="12G" --iodepth="4" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring_cmd" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_512_4_io_uring_cmd-reference_2.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_4_io_uring_cmd-reference_2.txt
io_uring_cmd-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=io_uring_cmd, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666790133,
  "timestamp_ms" : 1666790133702,
  "time" : "Wed Oct 26 15:15:33 2022",
  "jobs" : [
    {
      "jobname" : "io_uring_cmd-reference",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring_cmd-reference",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "4",
        "bs" : "512",
        "direct" : "0",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "io_uring_cmd",
        "filename" : "/dev/ng0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 1593121280,
        "io_kbytes" : 1555782,
        "bw_bytes" : 227556246,
        "bw" : 222222,
        "iops" : 444445.222111,
        "runtime" : 7001,
        "total_ios" : 3111561,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1162,
          "max" : 18786,
          "mean" : 1779.421280,
          "stddev" : 358.143700,
          "N" : 3111562
        },
        "clat_ns" : {
          "min" : 90,
          "max" : 126347,
          "mean" : 6962.273469,
          "stddev" : 2892.549055,
          "N" : 3111564,
          "percentile" : {
            "1.000000" : 4128,
            "5.000000" : 4640,
            "10.000000" : 4768,
            "20.000000" : 6112,
            "30.000000" : 6176,
            "40.000000" : 6304,
            "50.000000" : 6368,
            "60.000000" : 6496,
            "70.000000" : 6880,
            "80.000000" : 7456,
            "90.000000" : 8512,
            "95.000000" : 9152,
            "99.000000" : 20864,
            "99.500000" : 25472,
            "99.900000" : 27264,
            "99.950000" : 34048,
            "99.990000" : 68096
          }
        },
        "lat_ns" : {
          "min" : 5530,
          "max" : 127549,
          "mean" : 8741.694742,
          "stddev" : 2854.750029,
          "N" : 3111564
        },
        "bw_min" : 221009,
        "bw_max" : 223129,
        "bw_agg" : 100.000000,
        "bw_mean" : 222349.285714,
        "bw_dev" : 743.621536,
        "bw_samples" : 14,
        "iops_min" : 442018,
        "iops_max" : 446258,
        "iops_mean" : 444698.428571,
        "iops_stddev" : 1487.477820,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 16.157143,
      "sys_cpu" : 79.471429,
      "ctx" : 36179,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.010000,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.010000,
        "4" : 0.077582,
        "10" : 95.868408,
        "20" : 2.476474,
        "50" : 1.537267,
        "100" : 0.039498,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 4,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring_cmd-reference: (groupid=0, jobs=1): err= 0: pid=8456: Wed Oct 26 15:15:33 2022
  read: IOPS=444k, BW=217MiB/s (228MB/s)(1519MiB/7001msec)
    slat (nsec): min=1162, max=18786, avg=1779.42, stdev=358.14
    clat (nsec): min=90, max=126347, avg=6962.27, stdev=2892.55
     lat (usec): min=5, max=127, avg= 8.74, stdev= 2.85
    clat percentiles (nsec):
     |  1.00th=[ 4128],  5.00th=[ 4640], 10.00th=[ 4768], 20.00th=[ 6112],
     | 30.00th=[ 6176], 40.00th=[ 6304], 50.00th=[ 6368], 60.00th=[ 6496],
     | 70.00th=[ 6880], 80.00th=[ 7456], 90.00th=[ 8512], 95.00th=[ 9152],
     | 99.00th=[20864], 99.50th=[25472], 99.90th=[27264], 99.95th=[34048],
     | 99.99th=[68096]
   bw (  KiB/s): min=221009, max=223129, per=100.00%, avg=222349.29, stdev=743.62, samples=14
   iops        : min=442018, max=446258, avg=444698.43, stdev=1487.48, samples=14
  lat (nsec)   : 100=0.01%, 250=0.01%
  lat (usec)   : 2=0.01%, 4=0.08%, 10=95.87%, 20=2.48%, 50=1.54%
  lat (usec)   : 100=0.04%, 250=0.01%
  cpu          : usr=16.16%, sys=79.47%, ctx=36179, majf=0, minf=0
  IO depths    : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=3111561,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: bw=217MiB/s (228MB/s), 217MiB/s-217MiB/s (228MB/s-228MB/s), io=1519MiB (1593MB), run=7001-7001msec

$ rm /tmp/fio-output_512_4_io_uring_cmd-xnvme_0.txt
rm: cannot remove '/tmp/fio-output_512_4_io_uring_cmd-xnvme_0.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring_cmd-xnvme" --rw="randread" --size="12G" --iodepth="4" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring_cmd" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_512_4_io_uring_cmd-xnvme_0.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_1_io_uring_cmd-bdev_xnvme_0.txt
io_uring_cmd-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666789232,
  "timestamp_ms" : 1666789232866,
  "time" : "Wed Oct 26 15:00:32 2022",
  "jobs" : [
    {
      "jobname" : "io_uring_cmd-bdev_xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring_cmd-bdev_xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "1",
        "bs" : "512",
        "direct" : "0",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",
        "filename" : "Nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 584781824,
        "io_kbytes" : 571076,
        "bw_bytes" : 83528327,
        "bw" : 81570,
        "iops" : 163141.265533,
        "runtime" : 7001,
        "total_ios" : 1142152,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 230,
          "max" : 3617,
          "mean" : 269.127672,
          "stddev" : 21.625019,
          "N" : 1142152
        },
        "clat_ns" : {
          "min" : 2775,
          "max" : 111990,
          "mean" : 5644.366970,
          "stddev" : 1128.740165,
          "N" : 1142152,
          "percentile" : {
            "1.000000" : 5344,
            "5.000000" : 5344,
            "10.000000" : 5408,
            "20.000000" : 5408,
            "30.000000" : 5472,
            "40.000000" : 5472,
            "50.000000" : 5472,
            "60.000000" : 5536,
            "70.000000" : 5536,
            "80.000000" : 5600,
            "90.000000" : 5664,
            "95.000000" : 5728,
            "99.000000" : 11456,
            "99.500000" : 12480,
            "99.900000" : 16768,
            "99.950000" : 17024,
            "99.990000" : 58624
          }
        },
        "lat_ns" : {
          "min" : 5560,
          "max" : 112270,
          "mean" : 5913.494643,
          "stddev" : 1129.505716,
          "N" : 1142152
        },
        "bw_min" : 81452,
        "bw_max" : 81832,
        "bw_agg" : 100.000000,
        "bw_mean" : 81605.142857,
        "bw_dev" : 116.354139,
        "bw_samples" : 14,
        "iops_min" : 162905,
        "iops_max" : 163665,
        "iops_mean" : 163210.428571,
        "iops_stddev" : 232.757290,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 99.985714,
      "sys_cpu" : 0.000000,
      "ctx" : 13,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 100.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010000,
        "10" : 98.485491,
        "20" : 1.498575,
        "50" : 0.010000,
        "100" : 0.013221,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 1,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring_cmd-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=5227: Wed Oct 26 15:00:32 2022
  read: IOPS=163k, BW=79.7MiB/s (83.5MB/s)(558MiB/7001msec)
    slat (nsec): min=230, max=3617, avg=269.13, stdev=21.63
    clat (usec): min=2, max=111, avg= 5.64, stdev= 1.13
     lat (usec): min=5, max=112, avg= 5.91, stdev= 1.13
    clat percentiles (nsec):
     |  1.00th=[ 5344],  5.00th=[ 5344], 10.00th=[ 5408], 20.00th=[ 5408],
     | 30.00th=[ 5472], 40.00th=[ 5472], 50.00th=[ 5472], 60.00th=[ 5536],
     | 70.00th=[ 5536], 80.00th=[ 5600], 90.00th=[ 5664], 95.00th=[ 5728],
     | 99.00th=[11456], 99.50th=[12480], 99.90th=[16768], 99.95th=[17024],
     | 99.99th=[58624]
   bw (  KiB/s): min=81452, max=81832, per=100.00%, avg=81605.14, stdev=116.35, samples=14
   iops        : min=162905, max=163665, avg=163210.43, stdev=232.76, samples=14
  lat (usec)   : 4=0.01%, 10=98.49%, 20=1.50%, 50=0.01%, 100=0.01%
  lat (usec)   : 250=0.01%
  cpu          : usr=99.99%, sys=0.00%, ctx=13, majf=0, minf=0
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1142152,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=79.7MiB/s (83.5MB/s), 79.7MiB/s-79.7MiB/s (83.5MB/s-83.5MB/s), io=558MiB (585MB), run=7001-7001msec

$ cat /tmp/fio-output_512_4_io_uring_cmd-xnvme_0.txt
io_uring_cmd-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666790147,
  "timestamp_ms" : 1666790147445,
  "time" : "Wed Oct 26 15:15:47 2022",
  "jobs" : [
    {
      "jobname" : "io_uring_cmd-xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring_cmd-xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "4",
        "bs" : "512",
        "direct" : "0",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
        "xnvme_be" : "linux",
        "xnvme_async" : "io_uring_cmd",
        "xnvme_sync" : "nvme",
        "xnvme_admin" : "nvme",
        "xnvme_dev_nsid" : "1",
        "filename" : "/dev/ng0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 1479533568,
        "io_kbytes" : 1444857,
        "bw_bytes" : 211331748,
        "bw" : 206378,
        "iops" : 412756.891873,
        "runtime" : 7001,
        "total_ios" : 2889711,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1232,
          "max" : 22802,
          "mean" : 1856.760263,
          "stddev" : 376.862601,
          "N" : 2889711
        },
        "clat_ns" : {
          "min" : 140,
          "max" : 125245,
          "mean" : 7527.013691,
          "stddev" : 2833.703087,
          "N" : 2889714,
          "percentile" : {
            "1.000000" : 4320,
            "5.000000" : 4896,
            "10.000000" : 5792,
            "20.000000" : 6624,
            "30.000000" : 6688,
            "40.000000" : 7136,
            "50.000000" : 7264,
            "60.000000" : 7328,
            "70.000000" : 7520,
            "80.000000" : 7904,
            "90.000000" : 8512,
            "95.000000" : 9408,
            "99.000000" : 20864,
            "99.500000" : 25216,
            "99.900000" : 27520,
            "99.950000" : 34048,
            "99.990000" : 69120
          }
        },
        "lat_ns" : {
          "min" : 5701,
          "max" : 127048,
          "mean" : 9383.773646,
          "stddev" : 2814.443814,
          "N" : 2889714
        },
        "bw_min" : 205006,
        "bw_max" : 208277,
        "bw_agg" : 100.000000,
        "bw_mean" : 206571.076923,
        "bw_dev" : 1108.539088,
        "bw_samples" : 13,
        "iops_min" : 410012,
        "iops_max" : 416554,
        "iops_mean" : 413142.153846,
        "iops_stddev" : 2216.807195,
        "iops_samples" : 13
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 19.342857,
      "sys_cpu" : 76.642857,
      "ctx" : 35152,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010000,
        "10" : 95.738432,
        "20" : 2.739409,
        "50" : 1.478556,
        "100" : 0.041319,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 4,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring_cmd-xnvme: (groupid=0, jobs=1): err= 0: pid=8503: Wed Oct 26 15:15:47 2022
  read: IOPS=413k, BW=202MiB/s (211MB/s)(1411MiB/7001msec)
    slat (nsec): min=1232, max=22802, avg=1856.76, stdev=376.86
    clat (nsec): min=140, max=125245, avg=7527.01, stdev=2833.70
     lat (usec): min=5, max=127, avg= 9.38, stdev= 2.81
    clat percentiles (nsec):
     |  1.00th=[ 4320],  5.00th=[ 4896], 10.00th=[ 5792], 20.00th=[ 6624],
     | 30.00th=[ 6688], 40.00th=[ 7136], 50.00th=[ 7264], 60.00th=[ 7328],
     | 70.00th=[ 7520], 80.00th=[ 7904], 90.00th=[ 8512], 95.00th=[ 9408],
     | 99.00th=[20864], 99.50th=[25216], 99.90th=[27520], 99.95th=[34048],
     | 99.99th=[69120]
   bw (  KiB/s): min=205006, max=208277, per=100.00%, avg=206571.08, stdev=1108.54, samples=13
   iops        : min=410012, max=416554, avg=413142.15, stdev=2216.81, samples=13
  lat (nsec)   : 250=0.01%
  lat (usec)   : 4=0.01%, 10=95.74%, 20=2.74%, 50=1.48%, 100=0.04%
  lat (usec)   : 250=0.01%
  cpu          : usr=19.34%, sys=76.64%, ctx=35152, majf=0, minf=0
  IO depths    : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=2889711,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: bw=202MiB/s (211MB/s), 202MiB/s-202MiB/s (211MB/s-211MB/s), io=1411MiB (1480MB), run=7001-7001msec

$ rm /tmp/fio-output_512_4_io_uring_cmd-xnvme_1.txt
rm: cannot remove '/tmp/fio-output_512_4_io_uring_cmd-xnvme_1.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring_cmd-xnvme" --rw="randread" --size="12G" --iodepth="4" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring_cmd" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_512_4_io_uring_cmd-xnvme_1.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_4_io_uring_cmd-xnvme_1.txt
io_uring_cmd-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666790161,
  "timestamp_ms" : 1666790161185,
  "time" : "Wed Oct 26 15:16:01 2022",
  "jobs" : [
    {
      "jobname" : "io_uring_cmd-xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring_cmd-xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "4",
        "bs" : "512",
        "direct" : "0",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
        "xnvme_be" : "linux",
        "xnvme_async" : "io_uring_cmd",
        "xnvme_sync" : "nvme",
        "xnvme_admin" : "nvme",
        "xnvme_dev_nsid" : "1",
        "filename" : "/dev/ng0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 1513281024,
        "io_kbytes" : 1477813,
        "bw_bytes" : 216152124,
        "bw" : 211086,
        "iops" : 422171.546922,
        "runtime" : 7001,
        "total_ios" : 2955623,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1212,
          "max" : 68398,
          "mean" : 1804.931661,
          "stddev" : 383.353388,
          "N" : 2955624
        },
        "clat_ns" : {
          "min" : 130,
          "max" : 121648,
          "mean" : 7361.880692,
          "stddev" : 2801.249674,
          "N" : 2955626,
          "percentile" : {
            "1.000000" : 4256,
            "5.000000" : 4832,
            "10.000000" : 5088,
            "20.000000" : 6496,
            "30.000000" : 6560,
            "40.000000" : 6688,
            "50.000000" : 7072,
            "60.000000" : 7200,
            "70.000000" : 7392,
            "80.000000" : 7840,
            "90.000000" : 8384,
            "95.000000" : 9280,
            "99.000000" : 20864,
            "99.500000" : 24960,
            "99.900000" : 27264,
            "99.950000" : 33024,
            "99.990000" : 68096
          }
        },
        "lat_ns" : {
          "min" : 5770,
          "max" : 122910,
          "mean" : 9166.812258,
          "stddev" : 2786.079322,
          "N" : 2955626
        },
        "bw_min" : 209124,
        "bw_max" : 212490,
        "bw_agg" : 100.000000,
        "bw_mean" : 211123.571429,
        "bw_dev" : 1103.367274,
        "bw_samples" : 14,
        "iops_min" : 418248,
        "iops_max" : 424980,
        "iops_mean" : 422247.142857,
        "iops_stddev" : 2206.773449,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 19.828571,
      "sys_cpu" : 76.085714,
      "ctx" : 35312,
      "majf" : 0,
      "minf" : 1,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.010000,
        "4" : 0.010000,
        "10" : 95.890816,
        "20" : 2.646379,
        "50" : 1.422881,
        "100" : 0.037623,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 4,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring_cmd-xnvme: (groupid=0, jobs=1): err= 0: pid=8549: Wed Oct 26 15:16:01 2022
  read: IOPS=422k, BW=206MiB/s (216MB/s)(1443MiB/7001msec)
    slat (nsec): min=1212, max=68398, avg=1804.93, stdev=383.35
    clat (nsec): min=130, max=121648, avg=7361.88, stdev=2801.25
     lat (usec): min=5, max=122, avg= 9.17, stdev= 2.79
    clat percentiles (nsec):
     |  1.00th=[ 4256],  5.00th=[ 4832], 10.00th=[ 5088], 20.00th=[ 6496],
     | 30.00th=[ 6560], 40.00th=[ 6688], 50.00th=[ 7072], 60.00th=[ 7200],
     | 70.00th=[ 7392], 80.00th=[ 7840], 90.00th=[ 8384], 95.00th=[ 9280],
     | 99.00th=[20864], 99.50th=[24960], 99.90th=[27264], 99.95th=[33024],
     | 99.99th=[68096]
   bw (  KiB/s): min=209124, max=212490, per=100.00%, avg=211123.57, stdev=1103.37, samples=14
   iops        : min=418248, max=424980, avg=422247.14, stdev=2206.77, samples=14
  lat (nsec)   : 250=0.01%
  lat (usec)   : 2=0.01%, 4=0.01%, 10=95.89%, 20=2.65%, 50=1.42%
  lat (usec)   : 100=0.04%, 250=0.01%
  cpu          : usr=19.83%, sys=76.09%, ctx=35312, majf=0, minf=1
  IO depths    : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=2955623,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: bw=206MiB/s (216MB/s), 206MiB/s-206MiB/s (216MB/s-216MB/s), io=1443MiB (1513MB), run=7001-7001msec

$ rm /tmp/fio-output_512_4_io_uring_cmd-xnvme_2.txt
rm: cannot remove '/tmp/fio-output_512_4_io_uring_cmd-xnvme_2.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring_cmd-xnvme" --rw="randread" --size="12G" --iodepth="4" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring_cmd" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_512_4_io_uring_cmd-xnvme_2.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_4_io_uring_cmd-xnvme_2.txt
io_uring_cmd-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666790174,
  "timestamp_ms" : 1666790174990,
  "time" : "Wed Oct 26 15:16:14 2022",
  "jobs" : [
    {
      "jobname" : "io_uring_cmd-xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring_cmd-xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "4",
        "bs" : "512",
        "direct" : "0",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
        "xnvme_be" : "linux",
        "xnvme_async" : "io_uring_cmd",
        "xnvme_sync" : "nvme",
        "xnvme_admin" : "nvme",
        "xnvme_dev_nsid" : "1",
        "filename" : "/dev/ng0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 1493146112,
        "io_kbytes" : 1458150,
        "bw_bytes" : 213276119,
        "bw" : 208277,
        "iops" : 416554.349379,
        "runtime" : 7001,
        "total_ios" : 2916297,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1222,
          "max" : 23835,
          "mean" : 1835.246559,
          "stddev" : 380.983468,
          "N" : 2916298
        },
        "clat_ns" : {
          "min" : 140,
          "max" : 117631,
          "mean" : 7459.801219,
          "stddev" : 2783.673537,
          "N" : 2916300,
          "percentile" : {
            "1.000000" : 4256,
            "5.000000" : 4896,
            "10.000000" : 5792,
            "20.000000" : 6560,
            "30.000000" : 6624,
            "40.000000" : 6816,
            "50.000000" : 7200,
            "60.000000" : 7264,
            "70.000000" : 7520,
            "80.000000" : 7904,
            "90.000000" : 8384,
            "95.000000" : 9408,
            "99.000000" : 20864,
            "99.500000" : 25216,
            "99.900000" : 27264,
            "99.950000" : 33024,
            "99.990000" : 68096
          }
        },
        "lat_ns" : {
          "min" : 5740,
          "max" : 119665,
          "mean" : 9295.047879,
          "stddev" : 2765.898562,
          "N" : 2916300
        },
        "bw_min" : 206224,
        "bw_max" : 211144,
        "bw_agg" : 100.000000,
        "bw_mean" : 208366.357143,
        "bw_dev" : 1490.058678,
        "bw_samples" : 14,
        "iops_min" : 412448,
        "iops_max" : 422288,
        "iops_mean" : 416732.642857,
        "iops_stddev" : 2980.303175,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 20.971429,
      "sys_cpu" : 75.014286,
      "ctx" : 35185,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010000,
        "10" : 95.807011,
        "20" : 2.722768,
        "50" : 1.432056,
        "100" : 0.036450,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 4,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring_cmd-xnvme: (groupid=0, jobs=1): err= 0: pid=8596: Wed Oct 26 15:16:14 2022
  read: IOPS=417k, BW=203MiB/s (213MB/s)(1424MiB/7001msec)
    slat (nsec): min=1222, max=23835, avg=1835.25, stdev=380.98
    clat (nsec): min=140, max=117631, avg=7459.80, stdev=2783.67
     lat (usec): min=5, max=119, avg= 9.30, stdev= 2.77
    clat percentiles (nsec):
     |  1.00th=[ 4256],  5.00th=[ 4896], 10.00th=[ 5792], 20.00th=[ 6560],
     | 30.00th=[ 6624], 40.00th=[ 6816], 50.00th=[ 7200], 60.00th=[ 7264],
     | 70.00th=[ 7520], 80.00th=[ 7904], 90.00th=[ 8384], 95.00th=[ 9408],
     | 99.00th=[20864], 99.50th=[25216], 99.90th=[27264], 99.95th=[33024],
     | 99.99th=[68096]
   bw (  KiB/s): min=206224, max=211144, per=100.00%, avg=208366.36, stdev=1490.06, samples=14
   iops        : min=412448, max=422288, avg=416732.64, stdev=2980.30, samples=14
  lat (nsec)   : 250=0.01%
  lat (usec)   : 4=0.01%, 10=95.81%, 20=2.72%, 50=1.43%, 100=0.04%
  lat (usec)   : 250=0.01%
  cpu          : usr=20.97%, sys=75.01%, ctx=35185, majf=0, minf=0
  IO depths    : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=2916297,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: bw=203MiB/s (213MB/s), 203MiB/s-203MiB/s (213MB/s-213MB/s), io=1424MiB (1493MB), run=7001-7001msec

$ rm /tmp/fio-output_512_4_io_uring_cmd-bdev_xnvme_0.txt
rm: cannot remove '/tmp/fio-output_512_4_io_uring_cmd-bdev_xnvme_0.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring_cmd-bdev_xnvme" --rw="randread" --size="12G" --iodepth="4" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_4_io_uring_cmd-bdev_xnvme_0.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_512_4_io_uring_cmd-bdev_xnvme_0.txt
io_uring_cmd-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666790190,
  "timestamp_ms" : 1666790190135,
  "time" : "Wed Oct 26 15:16:30 2022",
  "jobs" : [
    {
      "jobname" : "io_uring_cmd-bdev_xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring_cmd-bdev_xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "4",
        "bs" : "512",
        "direct" : "0",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",
        "filename" : "Nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 1345384448,
        "io_kbytes" : 1313852,
        "bw_bytes" : 192170325,
        "bw" : 187666,
        "iops" : 375332.238252,
        "runtime" : 7001,
        "total_ios" : 2627701,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 651,
          "max" : 13565,
          "mean" : 938.106651,
          "stddev" : 69.869104,
          "N" : 2627701
        },
        "clat_ns" : {
          "min" : 1402,
          "max" : 108673,
          "mean" : 9369.322797,
          "stddev" : 2206.289705,
          "N" : 2627704,
          "percentile" : {
            "1.000000" : 7712,
            "5.000000" : 8256,
            "10.000000" : 8512,
            "20.000000" : 8640,
            "30.000000" : 8896,
            "40.000000" : 8896,
            "50.000000" : 9024,
            "60.000000" : 9152,
            "70.000000" : 9280,
            "80.000000" : 9408,
            "90.000000" : 9664,
            "95.000000" : 10688,
            "99.000000" : 20096,
            "99.500000" : 24192,
            "99.900000" : 29056,
            "99.950000" : 34048,
            "99.990000" : 67072
          }
        },
        "lat_ns" : {
          "min" : 7174,
          "max" : 109625,
          "mean" : 10307.429430,
          "stddev" : 2207.091785,
          "N" : 2627704
        },
        "bw_min" : 187307,
        "bw_max" : 188201,
        "bw_agg" : 100.000000,
        "bw_mean" : 187705.857143,
        "bw_dev" : 270.564327,
        "bw_samples" : 14,
        "iops_min" : 374614,
        "iops_max" : 376402,
        "iops_mean" : 375411.785714,
        "iops_stddev" : 541.060523,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 99.985714,
      "sys_cpu" : 0.000000,
      "ctx" : 12,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.100000,
        "2" : 3.338127,
        "4" : 96.661683,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.010000,
        "4" : 0.000000,
        "10" : 93.003199,
        "20" : 5.969895,
        "50" : 0.989154,
        "100" : 0.037714,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 4,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring_cmd-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=8655: Wed Oct 26 15:16:30 2022
  read: IOPS=375k, BW=183MiB/s (192MB/s)(1283MiB/7001msec)
    slat (nsec): min=651, max=13565, avg=938.11, stdev=69.87
    clat (nsec): min=1402, max=108673, avg=9369.32, stdev=2206.29
     lat (usec): min=7, max=109, avg=10.31, stdev= 2.21
    clat percentiles (nsec):
     |  1.00th=[ 7712],  5.00th=[ 8256], 10.00th=[ 8512], 20.00th=[ 8640],
     | 30.00th=[ 8896], 40.00th=[ 8896], 50.00th=[ 9024], 60.00th=[ 9152],
     | 70.00th=[ 9280], 80.00th=[ 9408], 90.00th=[ 9664], 95.00th=[10688],
     | 99.00th=[20096], 99.50th=[24192], 99.90th=[29056], 99.95th=[34048],
     | 99.99th=[67072]
   bw (  KiB/s): min=187307, max=188201, per=100.00%, avg=187705.86, stdev=270.56, samples=14
   iops        : min=374614, max=376402, avg=375411.79, stdev=541.06, samples=14
  lat (usec)   : 2=0.01%, 10=93.00%, 20=5.97%, 50=0.99%, 100=0.04%
  lat (usec)   : 250=0.01%
  cpu          : usr=99.99%, sys=0.00%, ctx=12, majf=0, minf=0
  IO depths    : 1=0.1%, 2=3.3%, 4=96.7%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=2627701,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: bw=183MiB/s (192MB/s), 183MiB/s-183MiB/s (192MB/s-192MB/s), io=1283MiB (1345MB), run=7001-7001msec

$ rm /tmp/fio-output_512_1_io_uring_cmd-bdev_xnvme_1.txt
rm: cannot remove '/tmp/fio-output_512_1_io_uring_cmd-bdev_xnvme_1.txt': No such file or directory

$ rm /tmp/fio-output_512_4_io_uring_cmd-bdev_xnvme_1.txt
rm: cannot remove '/tmp/fio-output_512_4_io_uring_cmd-bdev_xnvme_1.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring_cmd-bdev_xnvme" --rw="randread" --size="12G" --iodepth="4" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_4_io_uring_cmd-bdev_xnvme_1.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_512_4_io_uring_cmd-bdev_xnvme_1.txt
io_uring_cmd-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666790205,
  "timestamp_ms" : 1666790205340,
  "time" : "Wed Oct 26 15:16:45 2022",
  "jobs" : [
    {
      "jobname" : "io_uring_cmd-bdev_xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring_cmd-bdev_xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "4",
        "bs" : "512",
        "direct" : "0",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",
        "filename" : "Nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 1416632832,
        "io_kbytes" : 1383430,
        "bw_bytes" : 202347212,
        "bw" : 197604,
        "iops" : 395208.970147,
        "runtime" : 7001,
        "total_ios" : 2766858,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 171,
          "max" : 16481,
          "mean" : 269.504734,
          "stddev" : 26.227924,
          "N" : 2766858
        },
        "clat_ns" : {
          "min" : 380,
          "max" : 121328,
          "mean" : 9537.189593,
          "stddev" : 2328.868613,
          "N" : 2766861,
          "percentile" : {
            "1.000000" : 6432,
            "5.000000" : 7392,
            "10.000000" : 8032,
            "20.000000" : 8768,
            "30.000000" : 9024,
            "40.000000" : 9152,
            "50.000000" : 9280,
            "60.000000" : 9408,
            "70.000000" : 9536,
            "80.000000" : 9664,
            "90.000000" : 10304,
            "95.000000" : 11840,
            "99.000000" : 20096,
            "99.500000" : 21888,
            "99.900000" : 27264,
            "99.950000" : 33024,
            "99.990000" : 68096
          }
        },
        "lat_ns" : {
          "min" : 5581,
          "max" : 121598,
          "mean" : 9806.694310,
          "stddev" : 2330.569751,
          "N" : 2766861
        },
        "bw_min" : 197016,
        "bw_max" : 198564,
        "bw_agg" : 100.000000,
        "bw_mean" : 197684.571429,
        "bw_dev" : 480.434213,
        "bw_samples" : 14,
        "iops_min" : 394032,
        "iops_max" : 397128,
        "iops_mean" : 395369.142857,
        "iops_stddev" : 960.868426,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 99.985714,
      "sys_cpu" : 0.000000,
      "ctx" : 14,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.100000,
        "2" : 12.759744,
        "4" : 87.237184,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.010000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 87.316841,
        "20" : 11.658495,
        "50" : 0.982631,
        "100" : 0.041672,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 4,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring_cmd-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=8715: Wed Oct 26 15:16:45 2022
  read: IOPS=395k, BW=193MiB/s (202MB/s)(1351MiB/7001msec)
    slat (nsec): min=171, max=16481, avg=269.50, stdev=26.23
    clat (nsec): min=380, max=121328, avg=9537.19, stdev=2328.87
     lat (usec): min=5, max=121, avg= 9.81, stdev= 2.33
    clat percentiles (nsec):
     |  1.00th=[ 6432],  5.00th=[ 7392], 10.00th=[ 8032], 20.00th=[ 8768],
     | 30.00th=[ 9024], 40.00th=[ 9152], 50.00th=[ 9280], 60.00th=[ 9408],
     | 70.00th=[ 9536], 80.00th=[ 9664], 90.00th=[10304], 95.00th=[11840],
     | 99.00th=[20096], 99.50th=[21888], 99.90th=[27264], 99.95th=[33024],
     | 99.99th=[68096]
   bw (  KiB/s): min=197016, max=198564, per=100.00%, avg=197684.57, stdev=480.43, samples=14
   iops        : min=394032, max=397128, avg=395369.14, stdev=960.87, samples=14
  lat (nsec)   : 500=0.01%
  lat (usec)   : 10=87.32%, 20=11.66%, 50=0.98%, 100=0.04%, 250=0.01%
  cpu          : usr=99.99%, sys=0.00%, ctx=14, majf=0, minf=0
  IO depths    : 1=0.1%, 2=12.8%, 4=87.2%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=2766858,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: bw=193MiB/s (202MB/s), 193MiB/s-193MiB/s (202MB/s-202MB/s), io=1351MiB (1417MB), run=7001-7001msec

$ rm /tmp/fio-output_512_4_io_uring_cmd-bdev_xnvme_2.txt
rm: cannot remove '/tmp/fio-output_512_4_io_uring_cmd-bdev_xnvme_2.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring_cmd-bdev_xnvme" --rw="randread" --size="12G" --iodepth="4" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_4_io_uring_cmd-bdev_xnvme_2.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_512_4_io_uring_cmd-bdev_xnvme_2.txt
io_uring_cmd-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666790220,
  "timestamp_ms" : 1666790220567,
  "time" : "Wed Oct 26 15:17:00 2022",
  "jobs" : [
    {
      "jobname" : "io_uring_cmd-bdev_xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring_cmd-bdev_xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "4",
        "bs" : "512",
        "direct" : "0",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",
        "filename" : "Nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 1625583104,
        "io_kbytes" : 1587483,
        "bw_bytes" : 232192987,
        "bw" : 226750,
        "iops" : 453501.499786,
        "runtime" : 7001,
        "total_ios" : 3174964,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 200,
          "max" : 19697,
          "mean" : 283.869127,
          "stddev" : 75.463763,
          "N" : 3174964
        },
        "clat_ns" : {
          "min" : 501,
          "max" : 111770,
          "mean" : 8226.494525,
          "stddev" : 2228.476383,
          "N" : 3174967,
          "percentile" : {
            "1.000000" : 6368,
            "5.000000" : 6752,
            "10.000000" : 7008,
            "20.000000" : 7456,
            "30.000000" : 7584,
            "40.000000" : 7648,
            "50.000000" : 7776,
            "60.000000" : 7904,
            "70.000000" : 8096,
            "80.000000" : 8512,
            "90.000000" : 9280,
            "95.000000" : 10176,
            "99.000000" : 18816,
            "99.500000" : 21376,
            "99.900000" : 27520,
            "99.950000" : 33536,
            "99.990000" : 67072
          }
        },
        "lat_ns" : {
          "min" : 5861,
          "max" : 112050,
          "mean" : 8510.363694,
          "stddev" : 2229.327804,
          "N" : 3174967
        },
        "bw_min" : 226564,
        "bw_max" : 227133,
        "bw_agg" : 100.000000,
        "bw_mean" : 226841.785714,
        "bw_dev" : 143.797496,
        "bw_samples" : 14,
        "iops_min" : 453128,
        "iops_max" : 454266,
        "iops_mean" : 453683.642857,
        "iops_stddev" : 287.590416,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 99.985714,
      "sys_cpu" : 0.000000,
      "ctx" : 10,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.100000,
        "2" : 1.998952,
        "4" : 97.991001,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.010000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 94.702460,
        "20" : 4.585154,
        "50" : 0.674307,
        "100" : 0.037197,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 4,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring_cmd-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=8773: Wed Oct 26 15:17:00 2022
  read: IOPS=454k, BW=221MiB/s (232MB/s)(1550MiB/7001msec)
    slat (nsec): min=200, max=19697, avg=283.87, stdev=75.46
    clat (nsec): min=501, max=111770, avg=8226.49, stdev=2228.48
     lat (usec): min=5, max=112, avg= 8.51, stdev= 2.23
    clat percentiles (nsec):
     |  1.00th=[ 6368],  5.00th=[ 6752], 10.00th=[ 7008], 20.00th=[ 7456],
     | 30.00th=[ 7584], 40.00th=[ 7648], 50.00th=[ 7776], 60.00th=[ 7904],
     | 70.00th=[ 8096], 80.00th=[ 8512], 90.00th=[ 9280], 95.00th=[10176],
     | 99.00th=[18816], 99.50th=[21376], 99.90th=[27520], 99.95th=[33536],
     | 99.99th=[67072]
   bw (  KiB/s): min=226564, max=227133, per=100.00%, avg=226841.79, stdev=143.80, samples=14
   iops        : min=453128, max=454266, avg=453683.64, stdev=287.59, samples=14
  lat (nsec)   : 750=0.01%
  lat (usec)   : 10=94.70%, 20=4.59%, 50=0.67%, 100=0.04%, 250=0.01%
  cpu          : usr=99.99%, sys=0.00%, ctx=10, majf=0, minf=0
  IO depths    : 1=0.1%, 2=2.0%, 4=98.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=3174964,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: bw=221MiB/s (232MB/s), 221MiB/s-221MiB/s (232MB/s-232MB/s), io=1550MiB (1626MB), run=7001-7001msec

$ rm /tmp/fio-output_512_4_io_uring-reference_0.txt
rm: cannot remove '/tmp/fio-output_512_4_io_uring-reference_0.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring-reference" --rw="randread" --size="12G" --iodepth="4" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_4_io_uring-reference_0.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_4_io_uring-reference_0.txt
io_uring-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=io_uring, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666790234,
  "timestamp_ms" : 1666790234399,
  "time" : "Wed Oct 26 15:17:14 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-reference",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-reference",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "4",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "io_uring",
        "filename" : "/dev/nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 1530035200,
        "io_kbytes" : 1494175,
        "bw_bytes" : 218545236,
        "bw" : 213423,
        "iops" : 426845.593487,
        "runtime" : 7001,
        "total_ios" : 2988346,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1313,
          "max" : 23915,
          "mean" : 1844.950892,
          "stddev" : 353.846891,
          "N" : 2988347
        },
        "clat_ns" : {
          "min" : 110,
          "max" : 126256,
          "mean" : 7246.118301,
          "stddev" : 2909.869603,
          "N" : 2988349,
          "percentile" : {
            "1.000000" : 4192,
            "5.000000" : 4320,
            "10.000000" : 4768,
            "20.000000" : 6240,
            "30.000000" : 6496,
            "40.000000" : 6560,
            "50.000000" : 6752,
            "60.000000" : 7072,
            "70.000000" : 7520,
            "80.000000" : 8256,
            "90.000000" : 8768,
            "95.000000" : 9408,
            "99.000000" : 20608,
            "99.500000" : 24960,
            "99.900000" : 27008,
            "99.950000" : 33024,
            "99.990000" : 68096
          }
        },
        "lat_ns" : {
          "min" : 5380,
          "max" : 127619,
          "mean" : 9091.069296,
          "stddev" : 2889.364715,
          "N" : 2988349
        },
        "bw_min" : 212813,
        "bw_max" : 214359,
        "bw_agg" : 100.000000,
        "bw_mean" : 213525.857143,
        "bw_dev" : 374.802688,
        "bw_samples" : 14,
        "iops_min" : 425627,
        "iops_max" : 428718,
        "iops_mean" : 427051.928571,
        "iops_stddev" : 749.479559,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 17.014286,
      "sys_cpu" : 78.814286,
      "ctx" : 36626,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.010000,
        "500" : 0.010000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.010000,
        "4" : 0.010000,
        "10" : 95.847569,
        "20" : 2.632359,
        "50" : 1.478845,
        "100" : 0.039487,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 4,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ],
  "disk_util" : [
    {
      "name" : "nvme0n1",
      "read_ios" : 4302124,
      "write_ios" : 0,
      "read_merges" : 0,
      "write_merges" : 0,
      "read_ticks" : 23026,
      "write_ticks" : 0,
      "in_queue" : 23026,
      "util" : 99.088748
    }
  ]
}

io_uring-reference: (groupid=0, jobs=1): err= 0: pid=8823: Wed Oct 26 15:17:14 2022
  read: IOPS=427k, BW=208MiB/s (219MB/s)(1459MiB/7001msec)
    slat (nsec): min=1313, max=23915, avg=1844.95, stdev=353.85
    clat (nsec): min=110, max=126256, avg=7246.12, stdev=2909.87
     lat (usec): min=5, max=127, avg= 9.09, stdev= 2.89
    clat percentiles (nsec):
     |  1.00th=[ 4192],  5.00th=[ 4320], 10.00th=[ 4768], 20.00th=[ 6240],
     | 30.00th=[ 6496], 40.00th=[ 6560], 50.00th=[ 6752], 60.00th=[ 7072],
     | 70.00th=[ 7520], 80.00th=[ 8256], 90.00th=[ 8768], 95.00th=[ 9408],
     | 99.00th=[20608], 99.50th=[24960], 99.90th=[27008], 99.95th=[33024],
     | 99.99th=[68096]
   bw (  KiB/s): min=212813, max=214359, per=100.00%, avg=213525.86, stdev=374.80, samples=14
   iops        : min=425627, max=428718, avg=427051.93, stdev=749.48, samples=14
  lat (nsec)   : 250=0.01%, 500=0.01%
  lat (usec)   : 2=0.01%, 4=0.01%, 10=95.85%, 20=2.63%, 50=1.48%
  lat (usec)   : 100=0.04%, 250=0.01%
  cpu          : usr=17.01%, sys=78.81%, ctx=36626, majf=0, minf=0
  IO depths    : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=2988346,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: bw=208MiB/s (219MB/s), 208MiB/s-208MiB/s (219MB/s-219MB/s), io=1459MiB (1530MB), run=7001-7001msec

Disk stats (read/write):
  nvme0n1: ios=4302124/0, merge=0/0, ticks=23026/0, in_queue=23026, util=99.09%

$ rm /tmp/fio-output_512_4_io_uring-reference_1.txt
rm: cannot remove '/tmp/fio-output_512_4_io_uring-reference_1.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring_cmd-bdev_xnvme" --rw="randread" --size="12G" --iodepth="1" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_1_io_uring_cmd-bdev_xnvme_1.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created

$ /opt/fio/bin/fio --name="io_uring-reference" --rw="randread" --size="12G" --iodepth="4" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_4_io_uring-reference_1.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_4_io_uring-reference_1.txt
io_uring-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=io_uring, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666790248,
  "timestamp_ms" : 1666790248087,
  "time" : "Wed Oct 26 15:17:28 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-reference",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-reference",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "4",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "io_uring",
        "filename" : "/dev/nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 1519099904,
        "io_kbytes" : 1483496,
        "bw_bytes" : 216983274,
        "bw" : 211897,
        "iops" : 423794.886445,
        "runtime" : 7001,
        "total_ios" : 2966988,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1322,
          "max" : 22562,
          "mean" : 1861.858082,
          "stddev" : 352.891983,
          "N" : 2966989
        },
        "clat_ns" : {
          "min" : 100,
          "max" : 122830,
          "mean" : 7297.526278,
          "stddev" : 2887.510987,
          "N" : 2966991,
          "percentile" : {
            "1.000000" : 4192,
            "5.000000" : 4320,
            "10.000000" : 4832,
            "20.000000" : 6368,
            "30.000000" : 6560,
            "40.000000" : 6624,
            "50.000000" : 6880,
            "60.000000" : 7200,
            "70.000000" : 7520,
            "80.000000" : 8256,
            "90.000000" : 8768,
            "95.000000" : 9408,
            "99.000000" : 20608,
            "99.500000" : 24960,
            "99.900000" : 27008,
            "99.950000" : 33024,
            "99.990000" : 68096
          }
        },
        "lat_ns" : {
          "min" : 5731,
          "max" : 124212,
          "mean" : 9159.384584,
          "stddev" : 2868.348094,
          "N" : 2966991
        },
        "bw_min" : 211271,
        "bw_max" : 212946,
        "bw_agg" : 100.000000,
        "bw_mean" : 212027.500000,
        "bw_dev" : 425.605225,
        "bw_samples" : 14,
        "iops_min" : 422542,
        "iops_max" : 425892,
        "iops_mean" : 424055.357143,
        "iops_stddev" : 851.333308,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 17.200000,
      "sys_cpu" : 78.642857,
      "ctx" : 35976,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.010000,
        "4" : 0.010000,
        "10" : 95.813296,
        "20" : 2.761386,
        "50" : 1.384872,
        "100" : 0.039299,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 4,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ],
  "disk_util" : [
    {
      "name" : "nvme0n1",
      "read_ios" : 4273785,
      "write_ios" : 0,
      "read_merges" : 0,
      "write_merges" : 0,
      "read_ticks" : 22940,
      "write_ticks" : 0,
      "in_queue" : 22940,
      "util" : 99.098564
    }
  ]
}

io_uring-reference: (groupid=0, jobs=1): err= 0: pid=8869: Wed Oct 26 15:17:28 2022
  read: IOPS=424k, BW=207MiB/s (217MB/s)(1449MiB/7001msec)
    slat (nsec): min=1322, max=22562, avg=1861.86, stdev=352.89
    clat (nsec): min=100, max=122830, avg=7297.53, stdev=2887.51
     lat (usec): min=5, max=124, avg= 9.16, stdev= 2.87
    clat percentiles (nsec):
     |  1.00th=[ 4192],  5.00th=[ 4320], 10.00th=[ 4832], 20.00th=[ 6368],
     | 30.00th=[ 6560], 40.00th=[ 6624], 50.00th=[ 6880], 60.00th=[ 7200],
     | 70.00th=[ 7520], 80.00th=[ 8256], 90.00th=[ 8768], 95.00th=[ 9408],
     | 99.00th=[20608], 99.50th=[24960], 99.90th=[27008], 99.95th=[33024],
     | 99.99th=[68096]
   bw (  KiB/s): min=211271, max=212946, per=100.00%, avg=212027.50, stdev=425.61, samples=14
   iops        : min=422542, max=425892, avg=424055.36, stdev=851.33, samples=14
  lat (nsec)   : 250=0.01%
  lat (usec)   : 2=0.01%, 4=0.01%, 10=95.81%, 20=2.76%, 50=1.38%
  lat (usec)   : 100=0.04%, 250=0.01%
  cpu          : usr=17.20%, sys=78.64%, ctx=35976, majf=0, minf=0
  IO depths    : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=2966988,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: bw=207MiB/s (217MB/s), 207MiB/s-207MiB/s (217MB/s-217MB/s), io=1449MiB (1519MB), run=7001-7001msec

Disk stats (read/write):
  nvme0n1: ios=4273785/0, merge=0/0, ticks=22940/0, in_queue=22940, util=99.10%

$ rm /tmp/fio-output_512_4_io_uring-reference_2.txt
rm: cannot remove '/tmp/fio-output_512_4_io_uring-reference_2.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring-reference" --rw="randread" --size="12G" --iodepth="4" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_4_io_uring-reference_2.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_4_io_uring-reference_2.txt
io_uring-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=io_uring, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666790261,
  "timestamp_ms" : 1666790261828,
  "time" : "Wed Oct 26 15:17:41 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-reference",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-reference",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "4",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "io_uring",
        "filename" : "/dev/nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 1502555648,
        "io_kbytes" : 1467339,
        "bw_bytes" : 214620146,
        "bw" : 209589,
        "iops" : 419179.402942,
        "runtime" : 7001,
        "total_ios" : 2934675,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1333,
          "max" : 17603,
          "mean" : 1865.526482,
          "stddev" : 355.622282,
          "N" : 2934676
        },
        "clat_ns" : {
          "min" : 100,
          "max" : 118191,
          "mean" : 7382.369814,
          "stddev" : 2876.847571,
          "N" : 2934678,
          "percentile" : {
            "1.000000" : 4256,
            "5.000000" : 4320,
            "10.000000" : 4896,
            "20.000000" : 6496,
            "30.000000" : 6624,
            "40.000000" : 6752,
            "50.000000" : 7136,
            "60.000000" : 7264,
            "70.000000" : 7648,
            "80.000000" : 8256,
            "90.000000" : 8640,
            "95.000000" : 9408,
            "99.000000" : 20864,
            "99.500000" : 25216,
            "99.900000" : 27008,
            "99.950000" : 33024,
            "99.990000" : 68096
          }
        },
        "lat_ns" : {
          "min" : 5760,
          "max" : 119564,
          "mean" : 9247.896274,
          "stddev" : 2865.318682,
          "N" : 2934678
        },
        "bw_min" : 208949,
        "bw_max" : 210632,
        "bw_agg" : 100.000000,
        "bw_mean" : 209732.785714,
        "bw_dev" : 576.971694,
        "bw_samples" : 14,
        "iops_min" : 417898,
        "iops_max" : 421264,
        "iops_mean" : 419465.785714,
        "iops_stddev" : 1153.870889,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 17.242857,
      "sys_cpu" : 78.757143,
      "ctx" : 32783,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010000,
        "10" : 95.971786,
        "20" : 2.527912,
        "50" : 1.462308,
        "100" : 0.035949,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 4,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ],
  "disk_util" : [
    {
      "name" : "nvme0n1",
      "read_ios" : 4230620,
      "write_ios" : 0,
      "read_merges" : 0,
      "write_merges" : 0,
      "read_ticks" : 22672,
      "write_ticks" : 0,
      "in_queue" : 22672,
      "util" : 99.098564
    }
  ]
}

io_uring-reference: (groupid=0, jobs=1): err= 0: pid=8917: Wed Oct 26 15:17:41 2022
  read: IOPS=419k, BW=205MiB/s (215MB/s)(1433MiB/7001msec)
    slat (nsec): min=1333, max=17603, avg=1865.53, stdev=355.62
    clat (nsec): min=100, max=118191, avg=7382.37, stdev=2876.85
     lat (usec): min=5, max=119, avg= 9.25, stdev= 2.87
    clat percentiles (nsec):
     |  1.00th=[ 4256],  5.00th=[ 4320], 10.00th=[ 4896], 20.00th=[ 6496],
     | 30.00th=[ 6624], 40.00th=[ 6752], 50.00th=[ 7136], 60.00th=[ 7264],
     | 70.00th=[ 7648], 80.00th=[ 8256], 90.00th=[ 8640], 95.00th=[ 9408],
     | 99.00th=[20864], 99.50th=[25216], 99.90th=[27008], 99.95th=[33024],
     | 99.99th=[68096]
   bw (  KiB/s): min=208949, max=210632, per=100.00%, avg=209732.79, stdev=576.97, samples=14
   iops        : min=417898, max=421264, avg=419465.79, stdev=1153.87, samples=14
  lat (nsec)   : 250=0.01%
  lat (usec)   : 4=0.01%, 10=95.97%, 20=2.53%, 50=1.46%, 100=0.04%
  lat (usec)   : 250=0.01%
  cpu          : usr=17.24%, sys=78.76%, ctx=32783, majf=0, minf=0
  IO depths    : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=2934675,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: bw=205MiB/s (215MB/s), 205MiB/s-205MiB/s (215MB/s-215MB/s), io=1433MiB (1503MB), run=7001-7001msec

Disk stats (read/write):
  nvme0n1: ios=4230620/0, merge=0/0, ticks=22672/0, in_queue=22672, util=99.10%

$ rm /tmp/fio-output_512_4_io_uring-bdev_uring_0.txt
rm: cannot remove '/tmp/fio-output_512_4_io_uring-bdev_uring_0.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring-bdev_uring" --rw="randread" --size="12G" --iodepth="4" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_uring_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_4_io_uring-bdev_uring_0.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_512_4_io_uring-bdev_uring_0.txt
io_uring-bdev_uring: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666790276,
  "timestamp_ms" : 1666790276804,
  "time" : "Wed Oct 26 15:17:56 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-bdev_uring",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-bdev_uring",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "4",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_uring_foo.conf",
        "filename" : "Nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 1205324288,
        "io_kbytes" : 1177074,
        "bw_bytes" : 172164589,
        "bw" : 168129,
        "iops" : 336258.534495,
        "runtime" : 7001,
        "total_ios" : 2354146,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 110,
          "max" : 17002,
          "mean" : 179.156082,
          "stddev" : 149.002890,
          "N" : 2354146
        },
        "clat_ns" : {
          "min" : 5971,
          "max" : 115316,
          "mean" : 11061.462771,
          "stddev" : 3262.876327,
          "N" : 2354147,
          "percentile" : {
            "1.000000" : 6496,
            "5.000000" : 7328,
            "10.000000" : 7968,
            "20.000000" : 8384,
            "30.000000" : 8768,
            "40.000000" : 9152,
            "50.000000" : 10432,
            "60.000000" : 12096,
            "70.000000" : 12736,
            "80.000000" : 13248,
            "90.000000" : 13760,
            "95.000000" : 17280,
            "99.000000" : 21376,
            "99.500000" : 24192,
            "99.900000" : 30336,
            "99.950000" : 35072,
            "99.990000" : 66048
          }
        },
        "lat_ns" : {
          "min" : 6171,
          "max" : 115496,
          "mean" : 11240.618823,
          "stddev" : 3234.373994,
          "N" : 2354147
        },
        "bw_min" : 166257,
        "bw_max" : 170589,
        "bw_agg" : 100.000000,
        "bw_mean" : 168231.142857,
        "bw_dev" : 1493.829661,
        "bw_samples" : 14,
        "iops_min" : 332515,
        "iops_max" : 341178,
        "iops_mean" : 336462.428571,
        "iops_stddev" : 2987.544238,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 41.185714,
      "sys_cpu" : 58.785714,
      "ctx" : 11,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 6.693255,
        "2" : 43.215119,
        "4" : 50.091626,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 47.577253,
        "20" : 51.221589,
        "50" : 1.163182,
        "100" : 0.037848,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 4,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring-bdev_uring: (groupid=0, jobs=1): err= 0: pid=8974: Wed Oct 26 15:17:56 2022
  read: IOPS=336k, BW=164MiB/s (172MB/s)(1149MiB/7001msec)
    slat (nsec): min=110, max=17002, avg=179.16, stdev=149.00
    clat (usec): min=5, max=115, avg=11.06, stdev= 3.26
     lat (usec): min=6, max=115, avg=11.24, stdev= 3.23
    clat percentiles (nsec):
     |  1.00th=[ 6496],  5.00th=[ 7328], 10.00th=[ 7968], 20.00th=[ 8384],
     | 30.00th=[ 8768], 40.00th=[ 9152], 50.00th=[10432], 60.00th=[12096],
     | 70.00th=[12736], 80.00th=[13248], 90.00th=[13760], 95.00th=[17280],
     | 99.00th=[21376], 99.50th=[24192], 99.90th=[30336], 99.95th=[35072],
     | 99.99th=[66048]
   bw (  KiB/s): min=166257, max=170589, per=100.00%, avg=168231.14, stdev=1493.83, samples=14
   iops        : min=332515, max=341178, avg=336462.43, stdev=2987.54, samples=14
  lat (usec)   : 10=47.58%, 20=51.22%, 50=1.16%, 100=0.04%, 250=0.01%
  cpu          : usr=41.19%, sys=58.79%, ctx=11, majf=0, minf=0
  IO depths    : 1=6.7%, 2=43.2%, 4=50.1%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=2354146,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: bw=164MiB/s (172MB/s), 164MiB/s-164MiB/s (172MB/s-172MB/s), io=1149MiB (1205MB), run=7001-7001msec

$ rm /tmp/fio-output_512_4_io_uring-bdev_uring_1.txt
rm: cannot remove '/tmp/fio-output_512_4_io_uring-bdev_uring_1.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring-bdev_uring" --rw="randread" --size="12G" --iodepth="4" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_uring_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_4_io_uring-bdev_uring_1.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_512_1_io_uring_cmd-bdev_xnvme_1.txt
io_uring_cmd-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666789248,
  "timestamp_ms" : 1666789248165,
  "time" : "Wed Oct 26 15:00:48 2022",
  "jobs" : [
    {
      "jobname" : "io_uring_cmd-bdev_xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring_cmd-bdev_xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "1",
        "bs" : "512",
        "direct" : "0",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",
        "filename" : "Nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 407997952,
        "io_kbytes" : 398435,
        "bw_bytes" : 58277096,
        "bw" : 56911,
        "iops" : 113822.453935,
        "runtime" : 7001,
        "total_ios" : 796871,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 711,
          "max" : 20899,
          "mean" : 921.766521,
          "stddev" : 69.333618,
          "N" : 796871
        },
        "clat_ns" : {
          "min" : 5120,
          "max" : 80270,
          "mean" : 7528.237944,
          "stddev" : 1123.122707,
          "N" : 796871,
          "percentile" : {
            "1.000000" : 6880,
            "5.000000" : 7008,
            "10.000000" : 7072,
            "20.000000" : 7200,
            "30.000000" : 7264,
            "40.000000" : 7328,
            "50.000000" : 7392,
            "60.000000" : 7520,
            "70.000000" : 7584,
            "80.000000" : 7648,
            "90.000000" : 7776,
            "95.000000" : 7904,
            "99.000000" : 11712,
            "99.500000" : 14400,
            "99.900000" : 16512,
            "99.950000" : 17280,
            "99.990000" : 60672
          }
        },
        "lat_ns" : {
          "min" : 7153,
          "max" : 81202,
          "mean" : 8450.004465,
          "stddev" : 1126.519340,
          "N" : 796871
        },
        "bw_min" : 56840,
        "bw_max" : 57028,
        "bw_agg" : 100.000000,
        "bw_mean" : 56925.714286,
        "bw_dev" : 58.529977,
        "bw_samples" : 14,
        "iops_min" : 113680,
        "iops_max" : 114056,
        "iops_mean" : 113851.500000,
        "iops_stddev" : 117.153909,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 99.985714,
      "sys_cpu" : 0.000000,
      "ctx" : 11,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 100.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 98.200336,
        "20" : 1.771805,
        "50" : 0.010000,
        "100" : 0.018322,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 1,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring_cmd-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=5285: Wed Oct 26 15:00:48 2022
  read: IOPS=114k, BW=55.6MiB/s (58.3MB/s)(389MiB/7001msec)
    slat (nsec): min=711, max=20899, avg=921.77, stdev=69.33
    clat (nsec): min=5120, max=80270, avg=7528.24, stdev=1123.12
     lat (nsec): min=7153, max=81202, avg=8450.00, stdev=1126.52
    clat percentiles (nsec):
     |  1.00th=[ 6880],  5.00th=[ 7008], 10.00th=[ 7072], 20.00th=[ 7200],
     | 30.00th=[ 7264], 40.00th=[ 7328], 50.00th=[ 7392], 60.00th=[ 7520],
     | 70.00th=[ 7584], 80.00th=[ 7648], 90.00th=[ 7776], 95.00th=[ 7904],
     | 99.00th=[11712], 99.50th=[14400], 99.90th=[16512], 99.95th=[17280],
     | 99.99th=[60672]
   bw (  KiB/s): min=56840, max=57028, per=100.00%, avg=56925.71, stdev=58.53, samples=14
   iops        : min=113680, max=114056, avg=113851.50, stdev=117.15, samples=14
  lat (usec)   : 10=98.20%, 20=1.77%, 50=0.01%, 100=0.02%
  cpu          : usr=99.99%, sys=0.00%, ctx=11, majf=0, minf=0
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=796871,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=55.6MiB/s (58.3MB/s), 55.6MiB/s-55.6MiB/s (58.3MB/s-58.3MB/s), io=389MiB (408MB), run=7001-7001msec

$ cat /tmp/fio-output_512_4_io_uring-bdev_uring_1.txt
io_uring-bdev_uring: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666790291,
  "timestamp_ms" : 1666790291984,
  "time" : "Wed Oct 26 15:18:11 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-bdev_uring",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-bdev_uring",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "4",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_uring_foo.conf",
        "filename" : "Nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 1173137408,
        "io_kbytes" : 1145642,
        "bw_bytes" : 167591058,
        "bw" : 163663,
        "iops" : 327325.857143,
        "runtime" : 7000,
        "total_ios" : 2291281,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 120,
          "max" : 15419,
          "mean" : 174.303148,
          "stddev" : 129.262014,
          "N" : 2291281
        },
        "clat_ns" : {
          "min" : 6031,
          "max" : 97032,
          "mean" : 11358.835785,
          "stddev" : 3221.532516,
          "N" : 2291284,
          "percentile" : {
            "1.000000" : 6496,
            "5.000000" : 7392,
            "10.000000" : 8032,
            "20.000000" : 8512,
            "30.000000" : 8896,
            "40.000000" : 9664,
            "50.000000" : 11712,
            "60.000000" : 12608,
            "70.000000" : 12992,
            "80.000000" : 13376,
            "90.000000" : 13760,
            "95.000000" : 17536,
            "99.000000" : 21376,
            "99.500000" : 23936,
            "99.900000" : 29568,
            "99.950000" : 35072,
            "99.990000" : 66048
          }
        },
        "lat_ns" : {
          "min" : 6202,
          "max" : 97222,
          "mean" : 11533.138909,
          "stddev" : 3199.841005,
          "N" : 2291284
        },
        "bw_min" : 161805,
        "bw_max" : 165426,
        "bw_agg" : 100.000000,
        "bw_mean" : 163730.928571,
        "bw_dev" : 1280.287135,
        "bw_samples" : 14,
        "iops_min" : 323611,
        "iops_max" : 330852,
        "iops_mean" : 327461.928571,
        "iops_stddev" : 2560.458567,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 42.842857,
      "sys_cpu" : 57.128571,
      "ctx" : 16,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 9.001777,
        "2" : 44.955944,
        "4" : 46.042279,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 41.212099,
        "20" : 57.586259,
        "50" : 1.166116,
        "100" : 0.035657,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 4,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring-bdev_uring: (groupid=0, jobs=1): err= 0: pid=9031: Wed Oct 26 15:18:11 2022
  read: IOPS=327k, BW=160MiB/s (168MB/s)(1119MiB/7000msec)
    slat (nsec): min=120, max=15419, avg=174.30, stdev=129.26
    clat (nsec): min=6031, max=97032, avg=11358.84, stdev=3221.53
     lat (nsec): min=6202, max=97222, avg=11533.14, stdev=3199.84
    clat percentiles (nsec):
     |  1.00th=[ 6496],  5.00th=[ 7392], 10.00th=[ 8032], 20.00th=[ 8512],
     | 30.00th=[ 8896], 40.00th=[ 9664], 50.00th=[11712], 60.00th=[12608],
     | 70.00th=[12992], 80.00th=[13376], 90.00th=[13760], 95.00th=[17536],
     | 99.00th=[21376], 99.50th=[23936], 99.90th=[29568], 99.95th=[35072],
     | 99.99th=[66048]
   bw (  KiB/s): min=161805, max=165426, per=100.00%, avg=163730.93, stdev=1280.29, samples=14
   iops        : min=323611, max=330852, avg=327461.93, stdev=2560.46, samples=14
  lat (usec)   : 10=41.21%, 20=57.59%, 50=1.17%, 100=0.04%
  cpu          : usr=42.84%, sys=57.13%, ctx=16, majf=0, minf=0
  IO depths    : 1=9.0%, 2=45.0%, 4=46.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=2291281,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: bw=160MiB/s (168MB/s), 160MiB/s-160MiB/s (168MB/s-168MB/s), io=1119MiB (1173MB), run=7000-7000msec

$ rm /tmp/fio-output_512_4_io_uring-bdev_uring_2.txt
rm: cannot remove '/tmp/fio-output_512_4_io_uring-bdev_uring_2.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring-bdev_uring" --rw="randread" --size="12G" --iodepth="4" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_uring_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_4_io_uring-bdev_uring_2.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_512_4_io_uring-bdev_uring_2.txt
io_uring-bdev_uring: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666790307,
  "timestamp_ms" : 1666790307183,
  "time" : "Wed Oct 26 15:18:27 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-bdev_uring",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-bdev_uring",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "4",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_uring_foo.conf",
        "filename" : "Nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 1226106368,
        "io_kbytes" : 1197369,
        "bw_bytes" : 175133033,
        "bw" : 171028,
        "iops" : 342056.277675,
        "runtime" : 7001,
        "total_ios" : 2394736,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 110,
          "max" : 17543,
          "mean" : 182.469767,
          "stddev" : 147.807901,
          "N" : 2394736
        },
        "clat_ns" : {
          "min" : 5981,
          "max" : 134212,
          "mean" : 10884.014790,
          "stddev" : 3259.361616,
          "N" : 2394738,
          "percentile" : {
            "1.000000" : 6496,
            "5.000000" : 7392,
            "10.000000" : 7968,
            "20.000000" : 8384,
            "30.000000" : 8768,
            "40.000000" : 9024,
            "50.000000" : 9664,
            "60.000000" : 11712,
            "70.000000" : 12608,
            "80.000000" : 13120,
            "90.000000" : 13760,
            "95.000000" : 17536,
            "99.000000" : 21376,
            "99.500000" : 24192,
            "99.900000" : 30080,
            "99.950000" : 35072,
            "99.990000" : 67072
          }
        },
        "lat_ns" : {
          "min" : 6171,
          "max" : 134342,
          "mean" : 11066.484526,
          "stddev" : 3234.398550,
          "N" : 2394738
        },
        "bw_min" : 167302,
        "bw_max" : 174220,
        "bw_agg" : 100.000000,
        "bw_mean" : 171056.571429,
        "bw_dev" : 2202.020531,
        "bw_samples" : 14,
        "iops_min" : 334604,
        "iops_max" : 348440,
        "iops_mean" : 342113.142857,
        "iops_stddev" : 4404.041061,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 38.157143,
      "sys_cpu" : 61.814286,
      "ctx" : 13,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 5.818804,
        "2" : 42.767136,
        "4" : 51.414060,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 51.271330,
        "20" : 47.557309,
        "50" : 1.134739,
        "100" : 0.035996,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 4,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring-bdev_uring: (groupid=0, jobs=1): err= 0: pid=9088: Wed Oct 26 15:18:27 2022
  read: IOPS=342k, BW=167MiB/s (175MB/s)(1169MiB/7001msec)
    slat (nsec): min=110, max=17543, avg=182.47, stdev=147.81
    clat (usec): min=5, max=134, avg=10.88, stdev= 3.26
     lat (usec): min=6, max=134, avg=11.07, stdev= 3.23
    clat percentiles (nsec):
     |  1.00th=[ 6496],  5.00th=[ 7392], 10.00th=[ 7968], 20.00th=[ 8384],
     | 30.00th=[ 8768], 40.00th=[ 9024], 50.00th=[ 9664], 60.00th=[11712],
     | 70.00th=[12608], 80.00th=[13120], 90.00th=[13760], 95.00th=[17536],
     | 99.00th=[21376], 99.50th=[24192], 99.90th=[30080], 99.95th=[35072],
     | 99.99th=[67072]
   bw (  KiB/s): min=167302, max=174220, per=100.00%, avg=171056.57, stdev=2202.02, samples=14
   iops        : min=334604, max=348440, avg=342113.14, stdev=4404.04, samples=14
  lat (usec)   : 10=51.27%, 20=47.56%, 50=1.13%, 100=0.04%, 250=0.01%
  cpu          : usr=38.16%, sys=61.81%, ctx=13, majf=0, minf=0
  IO depths    : 1=5.8%, 2=42.8%, 4=51.4%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=2394736,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: bw=167MiB/s (175MB/s), 167MiB/s-167MiB/s (175MB/s-175MB/s), io=1169MiB (1226MB), run=7001-7001msec

$ rm /tmp/fio-output_512_4_io_uring-bdev_xnvme_0.txt
rm: cannot remove '/tmp/fio-output_512_4_io_uring-bdev_xnvme_0.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring-bdev_xnvme" --rw="randread" --size="12G" --iodepth="4" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_4_io_uring-bdev_xnvme_0.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_512_4_io_uring-bdev_xnvme_0.txt
io_uring-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666790322,
  "timestamp_ms" : 1666790322440,
  "time" : "Wed Oct 26 15:18:42 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-bdev_xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-bdev_xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "4",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring.conf",
        "filename" : "Nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 1559368192,
        "io_kbytes" : 1522820,
        "bw_bytes" : 222735065,
        "bw" : 217514,
        "iops" : 435028.853021,
        "runtime" : 7001,
        "total_ios" : 3045637,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1603,
          "max" : 68709,
          "mean" : 1767.973366,
          "stddev" : 121.396579,
          "N" : 3045638
        },
        "clat_ns" : {
          "min" : 220,
          "max" : 119765,
          "mean" : 7169.615502,
          "stddev" : 2186.549003,
          "N" : 3045640,
          "percentile" : {
            "1.000000" : 6624,
            "5.000000" : 6688,
            "10.000000" : 6688,
            "20.000000" : 6688,
            "30.000000" : 6752,
            "40.000000" : 6752,
            "50.000000" : 6752,
            "60.000000" : 6816,
            "70.000000" : 6816,
            "80.000000" : 6880,
            "90.000000" : 7072,
            "95.000000" : 7328,
            "99.000000" : 18816,
            "99.500000" : 21888,
            "99.900000" : 25216,
            "99.950000" : 32128,
            "99.990000" : 66048
          }
        },
        "lat_ns" : {
          "min" : 5411,
          "max" : 121538,
          "mean" : 8937.588861,
          "stddev" : 2192.642066,
          "N" : 3045640
        },
        "bw_min" : 216461,
        "bw_max" : 219449,
        "bw_agg" : 100.000000,
        "bw_mean" : 217555.357143,
        "bw_dev" : 811.925310,
        "bw_samples" : 14,
        "iops_min" : 432922,
        "iops_max" : 438898,
        "iops_mean" : 435110.714286,
        "iops_stddev" : 1623.850620,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 29.928571,
      "sys_cpu" : 70.042857,
      "ctx" : 17,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.010000,
        "500" : 0.010000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010000,
        "10" : 95.836306,
        "20" : 3.568055,
        "50" : 0.557814,
        "100" : 0.036511,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 4,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=9145: Wed Oct 26 15:18:42 2022
  read: IOPS=435k, BW=212MiB/s (223MB/s)(1487MiB/7001msec)
    slat (nsec): min=1603, max=68709, avg=1767.97, stdev=121.40
    clat (nsec): min=220, max=119765, avg=7169.62, stdev=2186.55
     lat (usec): min=5, max=121, avg= 8.94, stdev= 2.19
    clat percentiles (nsec):
     |  1.00th=[ 6624],  5.00th=[ 6688], 10.00th=[ 6688], 20.00th=[ 6688],
     | 30.00th=[ 6752], 40.00th=[ 6752], 50.00th=[ 6752], 60.00th=[ 6816],
     | 70.00th=[ 6816], 80.00th=[ 6880], 90.00th=[ 7072], 95.00th=[ 7328],
     | 99.00th=[18816], 99.50th=[21888], 99.90th=[25216], 99.95th=[32128],
     | 99.99th=[66048]
   bw (  KiB/s): min=216461, max=219449, per=100.00%, avg=217555.36, stdev=811.93, samples=14
   iops        : min=432922, max=438898, avg=435110.71, stdev=1623.85, samples=14
  lat (nsec)   : 250=0.01%, 500=0.01%
  lat (usec)   : 4=0.01%, 10=95.84%, 20=3.57%, 50=0.56%, 100=0.04%
  lat (usec)   : 250=0.01%
  cpu          : usr=29.93%, sys=70.04%, ctx=17, majf=0, minf=0
  IO depths    : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=3045637,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: bw=212MiB/s (223MB/s), 212MiB/s-212MiB/s (223MB/s-223MB/s), io=1487MiB (1559MB), run=7001-7001msec

$ rm /tmp/fio-output_512_4_io_uring-bdev_xnvme_1.txt
rm: cannot remove '/tmp/fio-output_512_4_io_uring-bdev_xnvme_1.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring-bdev_xnvme" --rw="randread" --size="12G" --iodepth="4" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_4_io_uring-bdev_xnvme_1.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_512_4_io_uring-bdev_xnvme_1.txt
io_uring-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666790337,
  "timestamp_ms" : 1666790337589,
  "time" : "Wed Oct 26 15:18:57 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-bdev_xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-bdev_xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "4",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring.conf",
        "filename" : "Nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 1562582528,
        "io_kbytes" : 1525959,
        "bw_bytes" : 223194190,
        "bw" : 217963,
        "iops" : 435925.582060,
        "runtime" : 7001,
        "total_ios" : 3051915,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1593,
          "max" : 23705,
          "mean" : 1775.126412,
          "stddev" : 116.995816,
          "N" : 3051916
        },
        "clat_ns" : {
          "min" : 231,
          "max" : 117450,
          "mean" : 7161.374734,
          "stddev" : 2248.409210,
          "N" : 3051918,
          "percentile" : {
            "1.000000" : 6624,
            "5.000000" : 6624,
            "10.000000" : 6688,
            "20.000000" : 6688,
            "30.000000" : 6752,
            "40.000000" : 6752,
            "50.000000" : 6752,
            "60.000000" : 6752,
            "70.000000" : 6816,
            "80.000000" : 6880,
            "90.000000" : 7072,
            "95.000000" : 7264,
            "99.000000" : 19328,
            "99.500000" : 24192,
            "99.900000" : 25728,
            "99.950000" : 32128,
            "99.990000" : 66048
          }
        },
        "lat_ns" : {
          "min" : 5370,
          "max" : 119194,
          "mean" : 8936.501126,
          "stddev" : 2254.690146,
          "N" : 3051918
        },
        "bw_min" : 216970,
        "bw_max" : 219040,
        "bw_agg" : 100.000000,
        "bw_mean" : 218076.500000,
        "bw_dev" : 597.526279,
        "bw_samples" : 14,
        "iops_min" : 433940,
        "iops_max" : 438080,
        "iops_mean" : 436153.071429,
        "iops_stddev" : 1194.975537,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 30.000000,
      "sys_cpu" : 69.985714,
      "ctx" : 10,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.010000,
        "500" : 0.010000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010000,
        "10" : 96.118994,
        "20" : 3.222043,
        "50" : 0.619414,
        "100" : 0.035486,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 4,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=9203: Wed Oct 26 15:18:57 2022
  read: IOPS=436k, BW=213MiB/s (223MB/s)(1490MiB/7001msec)
    slat (nsec): min=1593, max=23705, avg=1775.13, stdev=117.00
    clat (nsec): min=231, max=117450, avg=7161.37, stdev=2248.41
     lat (usec): min=5, max=119, avg= 8.94, stdev= 2.25
    clat percentiles (nsec):
     |  1.00th=[ 6624],  5.00th=[ 6624], 10.00th=[ 6688], 20.00th=[ 6688],
     | 30.00th=[ 6752], 40.00th=[ 6752], 50.00th=[ 6752], 60.00th=[ 6752],
     | 70.00th=[ 6816], 80.00th=[ 6880], 90.00th=[ 7072], 95.00th=[ 7264],
     | 99.00th=[19328], 99.50th=[24192], 99.90th=[25728], 99.95th=[32128],
     | 99.99th=[66048]
   bw (  KiB/s): min=216970, max=219040, per=100.00%, avg=218076.50, stdev=597.53, samples=14
   iops        : min=433940, max=438080, avg=436153.07, stdev=1194.98, samples=14
  lat (nsec)   : 250=0.01%, 500=0.01%
  lat (usec)   : 4=0.01%, 10=96.12%, 20=3.22%, 50=0.62%, 100=0.04%
  lat (usec)   : 250=0.01%
  cpu          : usr=30.00%, sys=69.99%, ctx=10, majf=0, minf=0
  IO depths    : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=3051915,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: bw=213MiB/s (223MB/s), 213MiB/s-213MiB/s (223MB/s-223MB/s), io=1490MiB (1563MB), run=7001-7001msec

$ rm /tmp/fio-output_512_1_io_uring_cmd-bdev_xnvme_2.txt
rm: cannot remove '/tmp/fio-output_512_1_io_uring_cmd-bdev_xnvme_2.txt': No such file or directory

$ rm /tmp/fio-output_512_4_io_uring-bdev_xnvme_2.txt
rm: cannot remove '/tmp/fio-output_512_4_io_uring-bdev_xnvme_2.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring-bdev_xnvme" --rw="randread" --size="12G" --iodepth="4" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_4_io_uring-bdev_xnvme_2.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_512_4_io_uring-bdev_xnvme_2.txt
io_uring-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666790352,
  "timestamp_ms" : 1666790352617,
  "time" : "Wed Oct 26 15:19:12 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-bdev_xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-bdev_xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "4",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring.conf",
        "filename" : "Nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 1559137792,
        "io_kbytes" : 1522595,
        "bw_bytes" : 222702155,
        "bw" : 217482,
        "iops" : 434964.576489,
        "runtime" : 7001,
        "total_ios" : 3045187,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1593,
          "max" : 20638,
          "mean" : 1801.881652,
          "stddev" : 112.241295,
          "N" : 3045188
        },
        "clat_ns" : {
          "min" : 211,
          "max" : 117270,
          "mean" : 7169.162602,
          "stddev" : 2248.200291,
          "N" : 3045190,
          "percentile" : {
            "1.000000" : 6624,
            "5.000000" : 6624,
            "10.000000" : 6688,
            "20.000000" : 6688,
            "30.000000" : 6752,
            "40.000000" : 6752,
            "50.000000" : 6752,
            "60.000000" : 6816,
            "70.000000" : 6816,
            "80.000000" : 6880,
            "90.000000" : 7072,
            "95.000000" : 7264,
            "99.000000" : 19328,
            "99.500000" : 24192,
            "99.900000" : 25728,
            "99.950000" : 32384,
            "99.990000" : 67072
          }
        },
        "lat_ns" : {
          "min" : 5360,
          "max" : 118993,
          "mean" : 8971.044281,
          "stddev" : 2253.640954,
          "N" : 3045190
        },
        "bw_min" : 216412,
        "bw_max" : 218927,
        "bw_agg" : 100.000000,
        "bw_mean" : 217521.785714,
        "bw_dev" : 661.670750,
        "bw_samples" : 14,
        "iops_min" : 432824,
        "iops_max" : 437854,
        "iops_mean" : 435043.642857,
        "iops_stddev" : 1323.496628,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 32.385714,
      "sys_cpu" : 67.600000,
      "ctx" : 14,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.010000,
        "500" : 0.010000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010000,
        "10" : 96.063559,
        "20" : 3.301308,
        "50" : 0.594709,
        "100" : 0.036911,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 4,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=9260: Wed Oct 26 15:19:12 2022
  read: IOPS=435k, BW=212MiB/s (223MB/s)(1487MiB/7001msec)
    slat (nsec): min=1593, max=20638, avg=1801.88, stdev=112.24
    clat (nsec): min=211, max=117270, avg=7169.16, stdev=2248.20
     lat (usec): min=5, max=118, avg= 8.97, stdev= 2.25
    clat percentiles (nsec):
     |  1.00th=[ 6624],  5.00th=[ 6624], 10.00th=[ 6688], 20.00th=[ 6688],
     | 30.00th=[ 6752], 40.00th=[ 6752], 50.00th=[ 6752], 60.00th=[ 6816],
     | 70.00th=[ 6816], 80.00th=[ 6880], 90.00th=[ 7072], 95.00th=[ 7264],
     | 99.00th=[19328], 99.50th=[24192], 99.90th=[25728], 99.95th=[32384],
     | 99.99th=[67072]
   bw (  KiB/s): min=216412, max=218927, per=100.00%, avg=217521.79, stdev=661.67, samples=14
   iops        : min=432824, max=437854, avg=435043.64, stdev=1323.50, samples=14
  lat (nsec)   : 250=0.01%, 500=0.01%
  lat (usec)   : 4=0.01%, 10=96.06%, 20=3.30%, 50=0.59%, 100=0.04%
  lat (usec)   : 250=0.01%
  cpu          : usr=32.39%, sys=67.60%, ctx=14, majf=0, minf=0
  IO depths    : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=3045187,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: bw=212MiB/s (223MB/s), 212MiB/s-212MiB/s (223MB/s-223MB/s), io=1487MiB (1559MB), run=7001-7001msec

$ rm /tmp/fio-output_512_4_io_uring-xnvme_0.txt
rm: cannot remove '/tmp/fio-output_512_4_io_uring-xnvme_0.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring-xnvme" --rw="randread" --size="12G" --iodepth="4" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_4_io_uring-xnvme_0.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_4_io_uring-xnvme_0.txt
io_uring-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666790366,
  "timestamp_ms" : 1666790366377,
  "time" : "Wed Oct 26 15:19:26 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "4",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
        "xnvme_be" : "linux",
        "xnvme_async" : "io_uring",
        "xnvme_sync" : "nvme",
        "xnvme_admin" : "nvme",
        "xnvme_dev_nsid" : "1",
        "filename" : "/dev/nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 1420984832,
        "io_kbytes" : 1387680,
        "bw_bytes" : 202968837,
        "bw" : 198211,
        "iops" : 396423.082417,
        "runtime" : 7001,
        "total_ios" : 2775358,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1382,
          "max" : 67266,
          "mean" : 1980.791384,
          "stddev" : 330.353282,
          "N" : 2775358
        },
        "clat_ns" : {
          "min" : 2054,
          "max" : 124162,
          "mean" : 7810.983699,
          "stddev" : 2690.900419,
          "N" : 2775361,
          "percentile" : {
            "1.000000" : 4448,
            "5.000000" : 5152,
            "10.000000" : 6816,
            "20.000000" : 6944,
            "30.000000" : 7392,
            "40.000000" : 7456,
            "50.000000" : 7456,
            "60.000000" : 7520,
            "70.000000" : 7648,
            "80.000000" : 8032,
            "90.000000" : 8384,
            "95.000000" : 9408,
            "99.000000" : 20864,
            "99.500000" : 25216,
            "99.900000" : 27264,
            "99.950000" : 33536,
            "99.990000" : 69120
          }
        },
        "lat_ns" : {
          "min" : 5781,
          "max" : 126286,
          "mean" : 9791.775021,
          "stddev" : 2666.740323,
          "N" : 2775361
        },
        "bw_min" : 197094,
        "bw_max" : 199345,
        "bw_agg" : 100.000000,
        "bw_mean" : 198313.214286,
        "bw_dev" : 631.854801,
        "bw_samples" : 14,
        "iops_min" : 394188,
        "iops_max" : 398690,
        "iops_mean" : 396626.571429,
        "iops_stddev" : 1263.775583,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 17.671429,
      "sys_cpu" : 78.628571,
      "ctx" : 27529,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010000,
        "10" : 96.050924,
        "20" : 2.666179,
        "50" : 1.239480,
        "100" : 0.041184,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 4,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring-xnvme: (groupid=0, jobs=1): err= 0: pid=9307: Wed Oct 26 15:19:26 2022
  read: IOPS=396k, BW=194MiB/s (203MB/s)(1355MiB/7001msec)
    slat (nsec): min=1382, max=67266, avg=1980.79, stdev=330.35
    clat (usec): min=2, max=124, avg= 7.81, stdev= 2.69
     lat (usec): min=5, max=126, avg= 9.79, stdev= 2.67
    clat percentiles (nsec):
     |  1.00th=[ 4448],  5.00th=[ 5152], 10.00th=[ 6816], 20.00th=[ 6944],
     | 30.00th=[ 7392], 40.00th=[ 7456], 50.00th=[ 7456], 60.00th=[ 7520],
     | 70.00th=[ 7648], 80.00th=[ 8032], 90.00th=[ 8384], 95.00th=[ 9408],
     | 99.00th=[20864], 99.50th=[25216], 99.90th=[27264], 99.95th=[33536],
     | 99.99th=[69120]
   bw (  KiB/s): min=197094, max=199345, per=100.00%, avg=198313.21, stdev=631.85, samples=14
   iops        : min=394188, max=398690, avg=396626.57, stdev=1263.78, samples=14
  lat (usec)   : 4=0.01%, 10=96.05%, 20=2.67%, 50=1.24%, 100=0.04%
  lat (usec)   : 250=0.01%
  cpu          : usr=17.67%, sys=78.63%, ctx=27529, majf=0, minf=0
  IO depths    : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=2775358,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: bw=194MiB/s (203MB/s), 194MiB/s-194MiB/s (203MB/s-203MB/s), io=1355MiB (1421MB), run=7001-7001msec

$ rm /tmp/fio-output_512_4_io_uring-xnvme_1.txt
rm: cannot remove '/tmp/fio-output_512_4_io_uring-xnvme_1.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring-xnvme" --rw="randread" --size="12G" --iodepth="4" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_4_io_uring-xnvme_1.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_4_io_uring-xnvme_1.txt
io_uring-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666790380,
  "timestamp_ms" : 1666790380127,
  "time" : "Wed Oct 26 15:19:40 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "4",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
        "xnvme_be" : "linux",
        "xnvme_async" : "io_uring",
        "xnvme_sync" : "nvme",
        "xnvme_admin" : "nvme",
        "xnvme_dev_nsid" : "1",
        "filename" : "/dev/nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 1402958336,
        "io_kbytes" : 1370076,
        "bw_bytes" : 200393991,
        "bw" : 195697,
        "iops" : 391393.943722,
        "runtime" : 7001,
        "total_ios" : 2740149,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1383,
          "max" : 22943,
          "mean" : 2020.769172,
          "stddev" : 287.667604,
          "N" : 2740150
        },
        "clat_ns" : {
          "min" : 2103,
          "max" : 122420,
          "mean" : 7908.474523,
          "stddev" : 2587.271863,
          "N" : 2740152,
          "percentile" : {
            "1.000000" : 4512,
            "5.000000" : 6816,
            "10.000000" : 6880,
            "20.000000" : 7456,
            "30.000000" : 7456,
            "40.000000" : 7520,
            "50.000000" : 7520,
            "60.000000" : 7584,
            "70.000000" : 7648,
            "80.000000" : 7840,
            "90.000000" : 8256,
            "95.000000" : 9024,
            "99.000000" : 20608,
            "99.500000" : 24960,
            "99.900000" : 27008,
            "99.950000" : 33024,
            "99.990000" : 69120
          }
        },
        "lat_ns" : {
          "min" : 5590,
          "max" : 123922,
          "mean" : 9929.243570,
          "stddev" : 2555.049304,
          "N" : 2740152
        },
        "bw_min" : 194510,
        "bw_max" : 197872,
        "bw_agg" : 100.000000,
        "bw_mean" : 195731.571429,
        "bw_dev" : 1047.071059,
        "bw_samples" : 14,
        "iops_min" : 389020,
        "iops_max" : 395744,
        "iops_mean" : 391463.214286,
        "iops_stddev" : 2094.197044,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 20.785714,
      "sys_cpu" : 75.628571,
      "ctx" : 27189,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010000,
        "10" : 96.072622,
        "20" : 2.739012,
        "50" : 1.149791,
        "100" : 0.037115,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 4,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring-xnvme: (groupid=0, jobs=1): err= 0: pid=9355: Wed Oct 26 15:19:40 2022
  read: IOPS=391k, BW=191MiB/s (200MB/s)(1338MiB/7001msec)
    slat (nsec): min=1383, max=22943, avg=2020.77, stdev=287.67
    clat (usec): min=2, max=122, avg= 7.91, stdev= 2.59
     lat (usec): min=5, max=123, avg= 9.93, stdev= 2.56
    clat percentiles (nsec):
     |  1.00th=[ 4512],  5.00th=[ 6816], 10.00th=[ 6880], 20.00th=[ 7456],
     | 30.00th=[ 7456], 40.00th=[ 7520], 50.00th=[ 7520], 60.00th=[ 7584],
     | 70.00th=[ 7648], 80.00th=[ 7840], 90.00th=[ 8256], 95.00th=[ 9024],
     | 99.00th=[20608], 99.50th=[24960], 99.90th=[27008], 99.95th=[33024],
     | 99.99th=[69120]
   bw (  KiB/s): min=194510, max=197872, per=100.00%, avg=195731.57, stdev=1047.07, samples=14
   iops        : min=389020, max=395744, avg=391463.21, stdev=2094.20, samples=14
  lat (usec)   : 4=0.01%, 10=96.07%, 20=2.74%, 50=1.15%, 100=0.04%
  lat (usec)   : 250=0.01%
  cpu          : usr=20.79%, sys=75.63%, ctx=27189, majf=0, minf=0
  IO depths    : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=2740149,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: bw=191MiB/s (200MB/s), 191MiB/s-191MiB/s (200MB/s-200MB/s), io=1338MiB (1403MB), run=7001-7001msec

$ rm /tmp/fio-output_512_4_io_uring-xnvme_2.txt
rm: cannot remove '/tmp/fio-output_512_4_io_uring-xnvme_2.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring_cmd-bdev_xnvme" --rw="randread" --size="12G" --iodepth="1" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_1_io_uring_cmd-bdev_xnvme_2.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created

$ /opt/fio/bin/fio --name="io_uring-xnvme" --rw="randread" --size="12G" --iodepth="4" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_4_io_uring-xnvme_2.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_4_io_uring-xnvme_2.txt
io_uring-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666790393,
  "timestamp_ms" : 1666790393982,
  "time" : "Wed Oct 26 15:19:53 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "4",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
        "xnvme_be" : "linux",
        "xnvme_async" : "io_uring",
        "xnvme_sync" : "nvme",
        "xnvme_admin" : "nvme",
        "xnvme_dev_nsid" : "1",
        "filename" : "/dev/nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 1408428544,
        "io_kbytes" : 1375418,
        "bw_bytes" : 201175338,
        "bw" : 196460,
        "iops" : 392920.011427,
        "runtime" : 7001,
        "total_ios" : 2750833,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1402,
          "max" : 43572,
          "mean" : 2012.219970,
          "stddev" : 307.976683,
          "N" : 2750834
        },
        "clat_ns" : {
          "min" : 131,
          "max" : 104275,
          "mean" : 7874.613087,
          "stddev" : 2621.702740,
          "N" : 2750836,
          "percentile" : {
            "1.000000" : 4448,
            "5.000000" : 6304,
            "10.000000" : 6880,
            "20.000000" : 7200,
            "30.000000" : 7456,
            "40.000000" : 7520,
            "50.000000" : 7520,
            "60.000000" : 7584,
            "70.000000" : 7648,
            "80.000000" : 7904,
            "90.000000" : 8256,
            "95.000000" : 9280,
            "99.000000" : 20864,
            "99.500000" : 25216,
            "99.900000" : 27008,
            "99.950000" : 33024,
            "99.990000" : 68096
          }
        },
        "lat_ns" : {
          "min" : 5771,
          "max" : 106169,
          "mean" : 9886.833346,
          "stddev" : 2594.294429,
          "N" : 2750836
        },
        "bw_min" : 195388,
        "bw_max" : 198951,
        "bw_agg" : 100.000000,
        "bw_mean" : 196495.071429,
        "bw_dev" : 1111.164217,
        "bw_samples" : 14,
        "iops_min" : 390776,
        "iops_max" : 397903,
        "iops_mean" : 392990.214286,
        "iops_stddev" : 2222.498462,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 21.085714,
      "sys_cpu" : 75.314286,
      "ctx" : 27181,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010000,
        "10" : 96.066900,
        "20" : 2.703181,
        "50" : 1.190040,
        "100" : 0.038025,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 4,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring-xnvme: (groupid=0, jobs=1): err= 0: pid=9402: Wed Oct 26 15:19:53 2022
  read: IOPS=393k, BW=192MiB/s (201MB/s)(1343MiB/7001msec)
    slat (nsec): min=1402, max=43572, avg=2012.22, stdev=307.98
    clat (nsec): min=131, max=104275, avg=7874.61, stdev=2621.70
     lat (usec): min=5, max=106, avg= 9.89, stdev= 2.59
    clat percentiles (nsec):
     |  1.00th=[ 4448],  5.00th=[ 6304], 10.00th=[ 6880], 20.00th=[ 7200],
     | 30.00th=[ 7456], 40.00th=[ 7520], 50.00th=[ 7520], 60.00th=[ 7584],
     | 70.00th=[ 7648], 80.00th=[ 7904], 90.00th=[ 8256], 95.00th=[ 9280],
     | 99.00th=[20864], 99.50th=[25216], 99.90th=[27008], 99.95th=[33024],
     | 99.99th=[68096]
   bw (  KiB/s): min=195388, max=198951, per=100.00%, avg=196495.07, stdev=1111.16, samples=14
   iops        : min=390776, max=397903, avg=392990.21, stdev=2222.50, samples=14
  lat (nsec)   : 250=0.01%
  lat (usec)   : 4=0.01%, 10=96.07%, 20=2.70%, 50=1.19%, 100=0.04%
  lat (usec)   : 250=0.01%
  cpu          : usr=21.09%, sys=75.31%, ctx=27181, majf=0, minf=0
  IO depths    : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=2750833,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: bw=192MiB/s (201MB/s), 192MiB/s-192MiB/s (201MB/s-201MB/s), io=1343MiB (1408MB), run=7001-7001msec

$ rm /tmp/fio-output_512_4_libaio-reference_0.txt
rm: cannot remove '/tmp/fio-output_512_4_libaio-reference_0.txt': No such file or directory

$ /opt/fio/bin/fio --name="libaio-reference" --rw="randread" --size="12G" --iodepth="4" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="libaio" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_4_libaio-reference_0.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_4_libaio-reference_0.txt
libaio-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=libaio, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666790407,
  "timestamp_ms" : 1666790407746,
  "time" : "Wed Oct 26 15:20:07 2022",
  "jobs" : [
    {
      "jobname" : "libaio-reference",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-reference",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "4",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "libaio",
        "filename" : "/dev/nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 1540268544,
        "io_kbytes" : 1504168,
        "bw_bytes" : 220006933,
        "bw" : 214850,
        "iops" : 429700.471361,
        "runtime" : 7001,
        "total_ios" : 3008333,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1442,
          "max" : 22633,
          "mean" : 1606.975883,
          "stddev" : 117.237640,
          "N" : 3008334
        },
        "clat_ns" : {
          "min" : 2945,
          "max" : 110618,
          "mean" : 7468.559953,
          "stddev" : 2665.426037,
          "N" : 3008336,
          "percentile" : {
            "1.000000" : 6816,
            "5.000000" : 6880,
            "10.000000" : 6880,
            "20.000000" : 6944,
            "30.000000" : 6944,
            "40.000000" : 6944,
            "50.000000" : 7008,
            "60.000000" : 7008,
            "70.000000" : 7072,
            "80.000000" : 7200,
            "90.000000" : 7264,
            "95.000000" : 7392,
            "99.000000" : 22400,
            "99.500000" : 27008,
            "99.900000" : 32640,
            "99.950000" : 35072,
            "99.990000" : 69120
          }
        },
        "lat_ns" : {
          "min" : 6051,
          "max" : 112181,
          "mean" : 9075.535866,
          "stddev" : 2669.379295,
          "N" : 3008336
        },
        "bw_min" : 213880,
        "bw_max" : 215909,
        "bw_agg" : 100.000000,
        "bw_mean" : 214911.214286,
        "bw_dev" : 609.982620,
        "bw_samples" : 14,
        "iops_min" : 427760,
        "iops_max" : 431818,
        "iops_mean" : 429822.428571,
        "iops_stddev" : 1219.965240,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 28.971429,
      "sys_cpu" : 67.200000,
      "ctx" : 26583,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010000,
        "10" : 96.583922,
        "20" : 2.229574,
        "50" : 1.152033,
        "100" : 0.034438,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 4,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ],
  "disk_util" : [
    {
      "name" : "nvme0n1",
      "read_ios" : 4333182,
      "write_ios" : 0,
      "read_merges" : 0,
      "write_merges" : 0,
      "read_ticks" : 23213,
      "write_ticks" : 0,
      "in_queue" : 23213,
      "util" : 99.069123
    }
  ]
}

libaio-reference: (groupid=0, jobs=1): err= 0: pid=9449: Wed Oct 26 15:20:07 2022
  read: IOPS=430k, BW=210MiB/s (220MB/s)(1469MiB/7001msec)
    slat (nsec): min=1442, max=22633, avg=1606.98, stdev=117.24
    clat (usec): min=2, max=110, avg= 7.47, stdev= 2.67
     lat (usec): min=6, max=112, avg= 9.08, stdev= 2.67
    clat percentiles (nsec):
     |  1.00th=[ 6816],  5.00th=[ 6880], 10.00th=[ 6880], 20.00th=[ 6944],
     | 30.00th=[ 6944], 40.00th=[ 6944], 50.00th=[ 7008], 60.00th=[ 7008],
     | 70.00th=[ 7072], 80.00th=[ 7200], 90.00th=[ 7264], 95.00th=[ 7392],
     | 99.00th=[22400], 99.50th=[27008], 99.90th=[32640], 99.95th=[35072],
     | 99.99th=[69120]
   bw (  KiB/s): min=213880, max=215909, per=100.00%, avg=214911.21, stdev=609.98, samples=14
   iops        : min=427760, max=431818, avg=429822.43, stdev=1219.97, samples=14
  lat (usec)   : 4=0.01%, 10=96.58%, 20=2.23%, 50=1.15%, 100=0.03%
  lat (usec)   : 250=0.01%
  cpu          : usr=28.97%, sys=67.20%, ctx=26583, majf=0, minf=0
  IO depths    : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=3008333,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: bw=210MiB/s (220MB/s), 210MiB/s-210MiB/s (220MB/s-220MB/s), io=1469MiB (1540MB), run=7001-7001msec

Disk stats (read/write):
  nvme0n1: ios=4333182/0, merge=0/0, ticks=23213/0, in_queue=23213, util=99.07%

$ rm /tmp/fio-output_512_4_libaio-reference_1.txt
rm: cannot remove '/tmp/fio-output_512_4_libaio-reference_1.txt': No such file or directory

$ /opt/fio/bin/fio --name="libaio-reference" --rw="randread" --size="12G" --iodepth="4" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="libaio" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_4_libaio-reference_1.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_4_libaio-reference_1.txt
libaio-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=libaio, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666790421,
  "timestamp_ms" : 1666790421560,
  "time" : "Wed Oct 26 15:20:21 2022",
  "jobs" : [
    {
      "jobname" : "libaio-reference",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-reference",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "4",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "libaio",
        "filename" : "/dev/nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 1547104768,
        "io_kbytes" : 1510844,
        "bw_bytes" : 220983397,
        "bw" : 215804,
        "iops" : 431607.627482,
        "runtime" : 7001,
        "total_ios" : 3021685,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1433,
          "max" : 30357,
          "mean" : 1591.780563,
          "stddev" : 110.756728,
          "N" : 3021686
        },
        "clat_ns" : {
          "min" : 4548,
          "max" : 126658,
          "mean" : 7446.293310,
          "stddev" : 2622.424351,
          "N" : 3021688,
          "percentile" : {
            "1.000000" : 6880,
            "5.000000" : 6880,
            "10.000000" : 6944,
            "20.000000" : 6944,
            "30.000000" : 6944,
            "40.000000" : 6944,
            "50.000000" : 7008,
            "60.000000" : 7008,
            "70.000000" : 7008,
            "80.000000" : 7072,
            "90.000000" : 7264,
            "95.000000" : 7392,
            "99.000000" : 21120,
            "99.500000" : 24960,
            "99.900000" : 31872,
            "99.950000" : 35072,
            "99.990000" : 70144
          }
        },
        "lat_ns" : {
          "min" : 6021,
          "max" : 128201,
          "mean" : 9038.073877,
          "stddev" : 2623.736564,
          "N" : 3021688
        },
        "bw_min" : 215056,
        "bw_max" : 217372,
        "bw_agg" : 100.000000,
        "bw_mean" : 215893.642857,
        "bw_dev" : 579.696420,
        "bw_samples" : 14,
        "iops_min" : 430112,
        "iops_max" : 434745,
        "iops_mean" : 431787.571429,
        "iops_stddev" : 1159.639978,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 32.028571,
      "sys_cpu" : 64.071429,
      "ctx" : 29587,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 96.336084,
        "20" : 2.499665,
        "50" : 1.126226,
        "100" : 0.037760,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 4,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ],
  "disk_util" : [
    {
      "name" : "nvme0n1",
      "read_ios" : 4350489,
      "write_ios" : 0,
      "read_merges" : 0,
      "write_merges" : 0,
      "read_ticks" : 21803,
      "write_ticks" : 0,
      "in_queue" : 21803,
      "util" : 99.078934
    }
  ]
}

libaio-reference: (groupid=0, jobs=1): err= 0: pid=9497: Wed Oct 26 15:20:21 2022
  read: IOPS=432k, BW=211MiB/s (221MB/s)(1475MiB/7001msec)
    slat (nsec): min=1433, max=30357, avg=1591.78, stdev=110.76
    clat (usec): min=4, max=126, avg= 7.45, stdev= 2.62
     lat (usec): min=6, max=128, avg= 9.04, stdev= 2.62
    clat percentiles (nsec):
     |  1.00th=[ 6880],  5.00th=[ 6880], 10.00th=[ 6944], 20.00th=[ 6944],
     | 30.00th=[ 6944], 40.00th=[ 6944], 50.00th=[ 7008], 60.00th=[ 7008],
     | 70.00th=[ 7008], 80.00th=[ 7072], 90.00th=[ 7264], 95.00th=[ 7392],
     | 99.00th=[21120], 99.50th=[24960], 99.90th=[31872], 99.95th=[35072],
     | 99.99th=[70144]
   bw (  KiB/s): min=215056, max=217372, per=100.00%, avg=215893.64, stdev=579.70, samples=14
   iops        : min=430112, max=434745, avg=431787.57, stdev=1159.64, samples=14
  lat (usec)   : 10=96.34%, 20=2.50%, 50=1.13%, 100=0.04%, 250=0.01%
  cpu          : usr=32.03%, sys=64.07%, ctx=29587, majf=0, minf=0
  IO depths    : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=3021685,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: bw=211MiB/s (221MB/s), 211MiB/s-211MiB/s (221MB/s-221MB/s), io=1475MiB (1547MB), run=7001-7001msec

Disk stats (read/write):
  nvme0n1: ios=4350489/0, merge=0/0, ticks=21803/0, in_queue=21803, util=99.08%

$ rm /tmp/fio-output_512_4_libaio-reference_2.txt
rm: cannot remove '/tmp/fio-output_512_4_libaio-reference_2.txt': No such file or directory

$ /opt/fio/bin/fio --name="libaio-reference" --rw="randread" --size="12G" --iodepth="4" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="libaio" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_4_libaio-reference_2.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_1_io_uring_cmd-bdev_xnvme_2.txt
io_uring_cmd-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666789263,
  "timestamp_ms" : 1666789263383,
  "time" : "Wed Oct 26 15:01:03 2022",
  "jobs" : [
    {
      "jobname" : "io_uring_cmd-bdev_xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring_cmd-bdev_xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "1",
        "bs" : "512",
        "direct" : "0",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",
        "filename" : "Nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 507951104,
        "io_kbytes" : 496046,
        "bw_bytes" : 72564443,
        "bw" : 70863,
        "iops" : 141727.428571,
        "runtime" : 7000,
        "total_ios" : 992092,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 250,
          "max" : 25638,
          "mean" : 278.643752,
          "stddev" : 68.503384,
          "N" : 992092
        },
        "clat_ns" : {
          "min" : 320,
          "max" : 78928,
          "mean" : 6484.535887,
          "stddev" : 1090.312041,
          "N" : 992092,
          "percentile" : {
            "1.000000" : 5984,
            "5.000000" : 6176,
            "10.000000" : 6240,
            "20.000000" : 6240,
            "30.000000" : 6304,
            "40.000000" : 6304,
            "50.000000" : 6368,
            "60.000000" : 6368,
            "70.000000" : 6432,
            "80.000000" : 6432,
            "90.000000" : 6496,
            "95.000000" : 6624,
            "99.000000" : 11456,
            "99.500000" : 13504,
            "99.900000" : 17024,
            "99.950000" : 19328,
            "99.990000" : 58112
          }
        },
        "lat_ns" : {
          "min" : 5991,
          "max" : 79199,
          "mean" : 6763.179639,
          "stddev" : 1092.110784,
          "N" : 992092
        },
        "bw_min" : 70687,
        "bw_max" : 71173,
        "bw_agg" : 100.000000,
        "bw_mean" : 70885.071429,
        "bw_dev" : 141.145023,
        "bw_samples" : 14,
        "iops_min" : 141374,
        "iops_max" : 142346,
        "iops_mean" : 141770.285714,
        "iops_stddev" : 282.276032,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 99.985714,
      "sys_cpu" : 0.000000,
      "ctx" : 10,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 100.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.010000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 98.826016,
        "20" : 1.129028,
        "50" : 0.031953,
        "100" : 0.012398,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 1,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring_cmd-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=5343: Wed Oct 26 15:01:03 2022
  read: IOPS=142k, BW=69.2MiB/s (72.6MB/s)(484MiB/7000msec)
    slat (nsec): min=250, max=25638, avg=278.64, stdev=68.50
    clat (nsec): min=320, max=78928, avg=6484.54, stdev=1090.31
     lat (nsec): min=5991, max=79199, avg=6763.18, stdev=1092.11
    clat percentiles (nsec):
     |  1.00th=[ 5984],  5.00th=[ 6176], 10.00th=[ 6240], 20.00th=[ 6240],
     | 30.00th=[ 6304], 40.00th=[ 6304], 50.00th=[ 6368], 60.00th=[ 6368],
     | 70.00th=[ 6432], 80.00th=[ 6432], 90.00th=[ 6496], 95.00th=[ 6624],
     | 99.00th=[11456], 99.50th=[13504], 99.90th=[17024], 99.95th=[19328],
     | 99.99th=[58112]
   bw (  KiB/s): min=70687, max=71173, per=100.00%, avg=70885.07, stdev=141.15, samples=14
   iops        : min=141374, max=142346, avg=141770.29, stdev=282.28, samples=14
  lat (nsec)   : 500=0.01%
  lat (usec)   : 10=98.83%, 20=1.13%, 50=0.03%, 100=0.01%
  cpu          : usr=99.99%, sys=0.00%, ctx=10, majf=0, minf=0
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=992092,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=69.2MiB/s (72.6MB/s), 69.2MiB/s-69.2MiB/s (72.6MB/s-72.6MB/s), io=484MiB (508MB), run=7000-7000msec

$ cat /tmp/fio-output_512_4_libaio-reference_2.txt
libaio-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=libaio, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666790435,
  "timestamp_ms" : 1666790435451,
  "time" : "Wed Oct 26 15:20:35 2022",
  "jobs" : [
    {
      "jobname" : "libaio-reference",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-reference",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "4",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "libaio",
        "filename" : "/dev/nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 1523316736,
        "io_kbytes" : 1487614,
        "bw_bytes" : 217585592,
        "bw" : 212485,
        "iops" : 424971.289816,
        "runtime" : 7001,
        "total_ios" : 2975224,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1453,
          "max" : 26480,
          "mean" : 1626.616954,
          "stddev" : 111.977939,
          "N" : 2975225
        },
        "clat_ns" : {
          "min" : 4588,
          "max" : 122941,
          "mean" : 7550.090039,
          "stddev" : 2701.313923,
          "N" : 2975227,
          "percentile" : {
            "1.000000" : 6944,
            "5.000000" : 6944,
            "10.000000" : 7008,
            "20.000000" : 7008,
            "30.000000" : 7008,
            "40.000000" : 7072,
            "50.000000" : 7072,
            "60.000000" : 7072,
            "70.000000" : 7136,
            "80.000000" : 7328,
            "90.000000" : 7392,
            "95.000000" : 7456,
            "99.000000" : 22400,
            "99.500000" : 27008,
            "99.900000" : 33024,
            "99.950000" : 38656,
            "99.990000" : 71168
          }
        },
        "lat_ns" : {
          "min" : 6101,
          "max" : 124514,
          "mean" : 9176.707014,
          "stddev" : 2705.260929,
          "N" : 2975227
        },
        "bw_min" : 212046,
        "bw_max" : 213823,
        "bw_agg" : 100.000000,
        "bw_mean" : 212606.928571,
        "bw_dev" : 548.418771,
        "bw_samples" : 14,
        "iops_min" : 424092,
        "iops_max" : 427646,
        "iops_mean" : 425213.928571,
        "iops_stddev" : 1096.881206,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 34.171429,
      "sys_cpu" : 62.028571,
      "ctx" : 26352,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 96.603953,
        "20" : 2.192776,
        "50" : 1.162669,
        "100" : 0.040568,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 4,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ],
  "disk_util" : [
    {
      "name" : "nvme0n1",
      "read_ios" : 4292890,
      "write_ios" : 0,
      "read_merges" : 0,
      "write_merges" : 0,
      "read_ticks" : 22927,
      "write_ticks" : 0,
      "in_queue" : 22927,
      "util" : 99.088748
    }
  ]
}

libaio-reference: (groupid=0, jobs=1): err= 0: pid=9543: Wed Oct 26 15:20:35 2022
  read: IOPS=425k, BW=208MiB/s (218MB/s)(1453MiB/7001msec)
    slat (nsec): min=1453, max=26480, avg=1626.62, stdev=111.98
    clat (usec): min=4, max=122, avg= 7.55, stdev= 2.70
     lat (usec): min=6, max=124, avg= 9.18, stdev= 2.71
    clat percentiles (nsec):
     |  1.00th=[ 6944],  5.00th=[ 6944], 10.00th=[ 7008], 20.00th=[ 7008],
     | 30.00th=[ 7008], 40.00th=[ 7072], 50.00th=[ 7072], 60.00th=[ 7072],
     | 70.00th=[ 7136], 80.00th=[ 7328], 90.00th=[ 7392], 95.00th=[ 7456],
     | 99.00th=[22400], 99.50th=[27008], 99.90th=[33024], 99.95th=[38656],
     | 99.99th=[71168]
   bw (  KiB/s): min=212046, max=213823, per=100.00%, avg=212606.93, stdev=548.42, samples=14
   iops        : min=424092, max=427646, avg=425213.93, stdev=1096.88, samples=14
  lat (usec)   : 10=96.60%, 20=2.19%, 50=1.16%, 100=0.04%, 250=0.01%
  cpu          : usr=34.17%, sys=62.03%, ctx=26352, majf=0, minf=0
  IO depths    : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=2975224,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: bw=208MiB/s (218MB/s), 208MiB/s-208MiB/s (218MB/s-218MB/s), io=1453MiB (1523MB), run=7001-7001msec

Disk stats (read/write):
  nvme0n1: ios=4292890/0, merge=0/0, ticks=22927/0, in_queue=22927, util=99.09%

$ rm /tmp/fio-output_512_4_libaio-bdev_aio_0.txt
rm: cannot remove '/tmp/fio-output_512_4_libaio-bdev_aio_0.txt': No such file or directory

$ /opt/fio/bin/fio --name="libaio-bdev_aio" --rw="randread" --size="12G" --iodepth="4" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_aio_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_4_libaio-bdev_aio_0.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_512_4_libaio-bdev_aio_0.txt
libaio-bdev_aio: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666790450,
  "timestamp_ms" : 1666790450686,
  "time" : "Wed Oct 26 15:20:50 2022",
  "jobs" : [
    {
      "jobname" : "libaio-bdev_aio",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-bdev_aio",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "4",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_aio_foo.conf",
        "filename" : "Nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 1562015232,
        "io_kbytes" : 1525405,
        "bw_bytes" : 223113159,
        "bw" : 217883,
        "iops" : 435767.318954,
        "runtime" : 7001,
        "total_ios" : 3050807,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1623,
          "max" : 24445,
          "mean" : 1823.933966,
          "stddev" : 122.188443,
          "N" : 3050808
        },
        "clat_ns" : {
          "min" : 271,
          "max" : 121558,
          "mean" : 6479.020192,
          "stddev" : 2349.697327,
          "N" : 3050807,
          "percentile" : {
            "1.000000" : 4384,
            "5.000000" : 4448,
            "10.000000" : 4576,
            "20.000000" : 4704,
            "30.000000" : 6496,
            "40.000000" : 6624,
            "50.000000" : 6688,
            "60.000000" : 6752,
            "70.000000" : 6752,
            "80.000000" : 6816,
            "90.000000" : 6944,
            "95.000000" : 7136,
            "99.000000" : 17536,
            "99.500000" : 20352,
            "99.900000" : 25472,
            "99.950000" : 30848,
            "99.990000" : 65280
          }
        },
        "lat_ns" : {
          "min" : 5370,
          "max" : 123371,
          "mean" : 8302.954171,
          "stddev" : 2352.126482,
          "N" : 3050807
        },
        "bw_min" : 216213,
        "bw_max" : 219913,
        "bw_agg" : 100.000000,
        "bw_mean" : 218014.214286,
        "bw_dev" : 1097.446559,
        "bw_samples" : 14,
        "iops_min" : 432426,
        "iops_max" : 439826,
        "iops_mean" : 436028.714286,
        "iops_stddev" : 2194.874043,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 38.500000,
      "sys_cpu" : 61.471429,
      "ctx" : 14,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.100000,
        "2" : 29.427558,
        "4" : 70.569525,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.010000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010000,
        "10" : 96.818940,
        "20" : 2.569976,
        "50" : 0.567129,
        "100" : 0.034483,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 4,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

libaio-bdev_aio: (groupid=0, jobs=1): err= 0: pid=9600: Wed Oct 26 15:20:50 2022
  read: IOPS=436k, BW=213MiB/s (223MB/s)(1490MiB/7001msec)
    slat (nsec): min=1623, max=24445, avg=1823.93, stdev=122.19
    clat (nsec): min=271, max=121558, avg=6479.02, stdev=2349.70
     lat (usec): min=5, max=123, avg= 8.30, stdev= 2.35
    clat percentiles (nsec):
     |  1.00th=[ 4384],  5.00th=[ 4448], 10.00th=[ 4576], 20.00th=[ 4704],
     | 30.00th=[ 6496], 40.00th=[ 6624], 50.00th=[ 6688], 60.00th=[ 6752],
     | 70.00th=[ 6752], 80.00th=[ 6816], 90.00th=[ 6944], 95.00th=[ 7136],
     | 99.00th=[17536], 99.50th=[20352], 99.90th=[25472], 99.95th=[30848],
     | 99.99th=[65280]
   bw (  KiB/s): min=216213, max=219913, per=100.00%, avg=218014.21, stdev=1097.45, samples=14
   iops        : min=432426, max=439826, avg=436028.71, stdev=2194.87, samples=14
  lat (nsec)   : 500=0.01%
  lat (usec)   : 4=0.01%, 10=96.82%, 20=2.57%, 50=0.57%, 100=0.03%
  lat (usec)   : 250=0.01%
  cpu          : usr=38.50%, sys=61.47%, ctx=14, majf=0, minf=0
  IO depths    : 1=0.1%, 2=29.4%, 4=70.6%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=3050807,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: bw=213MiB/s (223MB/s), 213MiB/s-213MiB/s (223MB/s-223MB/s), io=1490MiB (1562MB), run=7001-7001msec

$ rm /tmp/fio-output_512_4_libaio-bdev_aio_1.txt
rm: cannot remove '/tmp/fio-output_512_4_libaio-bdev_aio_1.txt': No such file or directory

$ /opt/fio/bin/fio --name="libaio-bdev_aio" --rw="randread" --size="12G" --iodepth="4" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_aio_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_4_libaio-bdev_aio_1.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_512_4_libaio-bdev_aio_1.txt
libaio-bdev_aio: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666790466,
  "timestamp_ms" : 1666790466054,
  "time" : "Wed Oct 26 15:21:06 2022",
  "jobs" : [
    {
      "jobname" : "libaio-bdev_aio",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-bdev_aio",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "4",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_aio_foo.conf",
        "filename" : "Nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 1556518912,
        "io_kbytes" : 1520038,
        "bw_bytes" : 222328083,
        "bw" : 217117,
        "iops" : 434234.109413,
        "runtime" : 7001,
        "total_ios" : 3040073,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1643,
          "max" : 40897,
          "mean" : 1833.893626,
          "stddev" : 124.135208,
          "N" : 3040073
        },
        "clat_ns" : {
          "min" : 290,
          "max" : 124533,
          "mean" : 6469.387454,
          "stddev" : 2382.210011,
          "N" : 3040076,
          "percentile" : {
            "1.000000" : 4448,
            "5.000000" : 4448,
            "10.000000" : 4576,
            "20.000000" : 4704,
            "30.000000" : 6432,
            "40.000000" : 6624,
            "50.000000" : 6688,
            "60.000000" : 6752,
            "70.000000" : 6816,
            "80.000000" : 6816,
            "90.000000" : 6944,
            "95.000000" : 7200,
            "99.000000" : 17536,
            "99.500000" : 20352,
            "99.900000" : 25728,
            "99.950000" : 31872,
            "99.990000" : 66048
          }
        },
        "lat_ns" : {
          "min" : 5440,
          "max" : 126347,
          "mean" : 8303.281079,
          "stddev" : 2384.467555,
          "N" : 3040076
        },
        "bw_min" : 216236,
        "bw_max" : 218096,
        "bw_agg" : 100.000000,
        "bw_mean" : 217165.357143,
        "bw_dev" : 551.447969,
        "bw_samples" : 14,
        "iops_min" : 432472,
        "iops_max" : 436190,
        "iops_mean" : 434330.785714,
        "iops_stddev" : 1102.928562,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 39.585714,
      "sys_cpu" : 60.385714,
      "ctx" : 14,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.100000,
        "2" : 31.062050,
        "4" : 68.935220,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.010000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.010000,
        "4" : 0.010000,
        "10" : 96.805800,
        "20" : 2.581681,
        "50" : 0.566894,
        "100" : 0.036775,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 4,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

libaio-bdev_aio: (groupid=0, jobs=1): err= 0: pid=9659: Wed Oct 26 15:21:06 2022
  read: IOPS=434k, BW=212MiB/s (222MB/s)(1484MiB/7001msec)
    slat (nsec): min=1643, max=40897, avg=1833.89, stdev=124.14
    clat (nsec): min=290, max=124533, avg=6469.39, stdev=2382.21
     lat (usec): min=5, max=126, avg= 8.30, stdev= 2.38
    clat percentiles (nsec):
     |  1.00th=[ 4448],  5.00th=[ 4448], 10.00th=[ 4576], 20.00th=[ 4704],
     | 30.00th=[ 6432], 40.00th=[ 6624], 50.00th=[ 6688], 60.00th=[ 6752],
     | 70.00th=[ 6816], 80.00th=[ 6816], 90.00th=[ 6944], 95.00th=[ 7200],
     | 99.00th=[17536], 99.50th=[20352], 99.90th=[25728], 99.95th=[31872],
     | 99.99th=[66048]
   bw (  KiB/s): min=216236, max=218096, per=100.00%, avg=217165.36, stdev=551.45, samples=14
   iops        : min=432472, max=436190, avg=434330.79, stdev=1102.93, samples=14
  lat (nsec)   : 500=0.01%
  lat (usec)   : 2=0.01%, 4=0.01%, 10=96.81%, 20=2.58%, 50=0.57%
  lat (usec)   : 100=0.04%, 250=0.01%
  cpu          : usr=39.59%, sys=60.39%, ctx=14, majf=0, minf=0
  IO depths    : 1=0.1%, 2=31.1%, 4=68.9%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=3040073,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: bw=212MiB/s (222MB/s), 212MiB/s-212MiB/s (222MB/s-222MB/s), io=1484MiB (1557MB), run=7001-7001msec

$ rm /tmp/fio-output_512_4_libaio-bdev_aio_2.txt
rm: cannot remove '/tmp/fio-output_512_4_libaio-bdev_aio_2.txt': No such file or directory

$ /opt/fio/bin/fio --name="libaio-bdev_aio" --rw="randread" --size="12G" --iodepth="4" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_aio_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_4_libaio-bdev_aio_2.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_512_4_libaio-bdev_aio_2.txt
libaio-bdev_aio: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666790481,
  "timestamp_ms" : 1666790481430,
  "time" : "Wed Oct 26 15:21:21 2022",
  "jobs" : [
    {
      "jobname" : "libaio-bdev_aio",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-bdev_aio",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "4",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_aio_foo.conf",
        "filename" : "Nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 1547431936,
        "io_kbytes" : 1511164,
        "bw_bytes" : 221030129,
        "bw" : 215849,
        "iops" : 431699.042994,
        "runtime" : 7001,
        "total_ios" : 3022325,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1643,
          "max" : 40075,
          "mean" : 1835.594190,
          "stddev" : 126.051482,
          "N" : 3022325
        },
        "clat_ns" : {
          "min" : 300,
          "max" : 120215,
          "mean" : 6500.137739,
          "stddev" : 2390.829741,
          "N" : 3022328,
          "percentile" : {
            "1.000000" : 4448,
            "5.000000" : 4512,
            "10.000000" : 4576,
            "20.000000" : 4704,
            "30.000000" : 6432,
            "40.000000" : 6688,
            "50.000000" : 6752,
            "60.000000" : 6752,
            "70.000000" : 6816,
            "80.000000" : 6880,
            "90.000000" : 7072,
            "95.000000" : 7264,
            "99.000000" : 17536,
            "99.500000" : 20608,
            "99.900000" : 25984,
            "99.950000" : 32128,
            "99.990000" : 64768
          }
        },
        "lat_ns" : {
          "min" : 5340,
          "max" : 122139,
          "mean" : 8335.731884,
          "stddev" : 2394.300500,
          "N" : 3022328
        },
        "bw_min" : 215090,
        "bw_max" : 216619,
        "bw_agg" : 100.000000,
        "bw_mean" : 215940.769231,
        "bw_dev" : 480.123969,
        "bw_samples" : 13,
        "iops_min" : 430180,
        "iops_max" : 433238,
        "iops_mean" : 431881.461538,
        "iops_stddev" : 960.297056,
        "iops_samples" : 13
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 40.442857,
      "sys_cpu" : 59.528571,
      "ctx" : 15,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.100000,
        "2" : 31.311093,
        "4" : 68.686789,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.010000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010000,
        "10" : 96.840710,
        "20" : 2.537020,
        "50" : 0.573764,
        "100" : 0.037653,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 4,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

libaio-bdev_aio: (groupid=0, jobs=1): err= 0: pid=9716: Wed Oct 26 15:21:21 2022
  read: IOPS=432k, BW=211MiB/s (221MB/s)(1476MiB/7001msec)
    slat (nsec): min=1643, max=40075, avg=1835.59, stdev=126.05
    clat (nsec): min=300, max=120215, avg=6500.14, stdev=2390.83
     lat (usec): min=5, max=122, avg= 8.34, stdev= 2.39
    clat percentiles (nsec):
     |  1.00th=[ 4448],  5.00th=[ 4512], 10.00th=[ 4576], 20.00th=[ 4704],
     | 30.00th=[ 6432], 40.00th=[ 6688], 50.00th=[ 6752], 60.00th=[ 6752],
     | 70.00th=[ 6816], 80.00th=[ 6880], 90.00th=[ 7072], 95.00th=[ 7264],
     | 99.00th=[17536], 99.50th=[20608], 99.90th=[25984], 99.95th=[32128],
     | 99.99th=[64768]
   bw (  KiB/s): min=215090, max=216619, per=100.00%, avg=215940.77, stdev=480.12, samples=13
   iops        : min=430180, max=433238, avg=431881.46, stdev=960.30, samples=13
  lat (nsec)   : 500=0.01%
  lat (usec)   : 4=0.01%, 10=96.84%, 20=2.54%, 50=0.57%, 100=0.04%
  lat (usec)   : 250=0.01%
  cpu          : usr=40.44%, sys=59.53%, ctx=15, majf=0, minf=0
  IO depths    : 1=0.1%, 2=31.3%, 4=68.7%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=3022325,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: bw=211MiB/s (221MB/s), 211MiB/s-211MiB/s (221MB/s-221MB/s), io=1476MiB (1547MB), run=7001-7001msec

$ rm /tmp/fio-output_512_1_io_uring-reference_0.txt
rm: cannot remove '/tmp/fio-output_512_1_io_uring-reference_0.txt': No such file or directory

$ rm /tmp/fio-output_512_4_libaio-bdev_xnvme_0.txt
rm: cannot remove '/tmp/fio-output_512_4_libaio-bdev_xnvme_0.txt': No such file or directory

$ /opt/fio/bin/fio --name="libaio-bdev_xnvme" --rw="randread" --size="12G" --iodepth="4" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_4_libaio-bdev_xnvme_0.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_512_4_libaio-bdev_xnvme_0.txt
libaio-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666790496,
  "timestamp_ms" : 1666790496711,
  "time" : "Wed Oct 26 15:21:36 2022",
  "jobs" : [
    {
      "jobname" : "libaio-bdev_xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-bdev_xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "4",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_xnvme_libaio.conf",
        "filename" : "Nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 1444080640,
        "io_kbytes" : 1410235,
        "bw_bytes" : 206267767,
        "bw" : 201433,
        "iops" : 402866.161977,
        "runtime" : 7001,
        "total_ios" : 2820466,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1673,
          "max" : 34555,
          "mean" : 1859.775083,
          "stddev" : 123.879869,
          "N" : 2820467
        },
        "clat_ns" : {
          "min" : 701,
          "max" : 115096,
          "mean" : 6746.611971,
          "stddev" : 2343.713145,
          "N" : 2820466,
          "percentile" : {
            "1.000000" : 5216,
            "5.000000" : 5280,
            "10.000000" : 5280,
            "20.000000" : 5344,
            "30.000000" : 5408,
            "40.000000" : 5472,
            "50.000000" : 7264,
            "60.000000" : 7328,
            "70.000000" : 7392,
            "80.000000" : 7456,
            "90.000000" : 7584,
            "95.000000" : 7968,
            "99.000000" : 17536,
            "99.500000" : 20608,
            "99.900000" : 26240,
            "99.950000" : 32128,
            "99.990000" : 63744
          }
        },
        "lat_ns" : {
          "min" : 5801,
          "max" : 116919,
          "mean" : 8606.387073,
          "stddev" : 2346.267108,
          "N" : 2820466
        },
        "bw_min" : 200600,
        "bw_max" : 202229,
        "bw_agg" : 100.000000,
        "bw_mean" : 201469.857143,
        "bw_dev" : 500.196093,
        "bw_samples" : 14,
        "iops_min" : 401200,
        "iops_max" : 404458,
        "iops_mean" : 402939.714286,
        "iops_stddev" : 1000.392187,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 37.757143,
      "sys_cpu" : 62.214286,
      "ctx" : 14,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.100000,
        "2" : 49.370813,
        "4" : 50.626563,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.010000,
        "1000" : 0.010000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010000,
        "10" : 96.743020,
        "20" : 2.692959,
        "50" : 0.519382,
        "100" : 0.038150,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 4,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

libaio-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=9774: Wed Oct 26 15:21:36 2022
  read: IOPS=403k, BW=197MiB/s (206MB/s)(1377MiB/7001msec)
    slat (nsec): min=1673, max=34555, avg=1859.78, stdev=123.88
    clat (nsec): min=701, max=115096, avg=6746.61, stdev=2343.71
     lat (usec): min=5, max=116, avg= 8.61, stdev= 2.35
    clat percentiles (nsec):
     |  1.00th=[ 5216],  5.00th=[ 5280], 10.00th=[ 5280], 20.00th=[ 5344],
     | 30.00th=[ 5408], 40.00th=[ 5472], 50.00th=[ 7264], 60.00th=[ 7328],
     | 70.00th=[ 7392], 80.00th=[ 7456], 90.00th=[ 7584], 95.00th=[ 7968],
     | 99.00th=[17536], 99.50th=[20608], 99.90th=[26240], 99.95th=[32128],
     | 99.99th=[63744]
   bw (  KiB/s): min=200600, max=202229, per=100.00%, avg=201469.86, stdev=500.20, samples=14
   iops        : min=401200, max=404458, avg=402939.71, stdev=1000.39, samples=14
  lat (nsec)   : 750=0.01%, 1000=0.01%
  lat (usec)   : 4=0.01%, 10=96.74%, 20=2.69%, 50=0.52%, 100=0.04%
  lat (usec)   : 250=0.01%
  cpu          : usr=37.76%, sys=62.21%, ctx=14, majf=0, minf=0
  IO depths    : 1=0.1%, 2=49.4%, 4=50.6%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=2820466,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: bw=197MiB/s (206MB/s), 197MiB/s-197MiB/s (206MB/s-206MB/s), io=1377MiB (1444MB), run=7001-7001msec

$ rm /tmp/fio-output_512_4_libaio-bdev_xnvme_1.txt
rm: cannot remove '/tmp/fio-output_512_4_libaio-bdev_xnvme_1.txt': No such file or directory

$ /opt/fio/bin/fio --name="libaio-bdev_xnvme" --rw="randread" --size="12G" --iodepth="4" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_4_libaio-bdev_xnvme_1.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_512_4_libaio-bdev_xnvme_1.txt
libaio-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666790512,
  "timestamp_ms" : 1666790512099,
  "time" : "Wed Oct 26 15:21:52 2022",
  "jobs" : [
    {
      "jobname" : "libaio-bdev_xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-bdev_xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "4",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_xnvme_libaio.conf",
        "filename" : "Nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 1448945664,
        "io_kbytes" : 1414986,
        "bw_bytes" : 206962671,
        "bw" : 202111,
        "iops" : 404223.539494,
        "runtime" : 7001,
        "total_ios" : 2829969,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1653,
          "max" : 22372,
          "mean" : 1839.170135,
          "stddev" : 121.423768,
          "N" : 2829970
        },
        "clat_ns" : {
          "min" : 691,
          "max" : 120626,
          "mean" : 6717.754872,
          "stddev" : 2347.788874,
          "N" : 2829972,
          "percentile" : {
            "1.000000" : 5152,
            "5.000000" : 5280,
            "10.000000" : 5280,
            "20.000000" : 5280,
            "30.000000" : 5344,
            "40.000000" : 5408,
            "50.000000" : 7264,
            "60.000000" : 7328,
            "70.000000" : 7328,
            "80.000000" : 7392,
            "90.000000" : 7584,
            "95.000000" : 7968,
            "99.000000" : 17536,
            "99.500000" : 20608,
            "99.900000" : 26240,
            "99.950000" : 32128,
            "99.990000" : 64256
          }
        },
        "lat_ns" : {
          "min" : 5520,
          "max" : 122500,
          "mean" : 8556.924986,
          "stddev" : 2350.299146,
          "N" : 2829972
        },
        "bw_min" : 201099,
        "bw_max" : 203576,
        "bw_agg" : 100.000000,
        "bw_mean" : 202148.357143,
        "bw_dev" : 804.850259,
        "bw_samples" : 14,
        "iops_min" : 402198,
        "iops_max" : 407152,
        "iops_mean" : 404296.714286,
        "iops_stddev" : 1609.700518,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 39.814286,
      "sys_cpu" : 60.157143,
      "ctx" : 12,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.100000,
        "2" : 49.212165,
        "4" : 50.785291,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.010000,
        "1000" : 0.010000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010000,
        "10" : 96.734523,
        "20" : 2.704376,
        "50" : 0.516825,
        "100" : 0.037209,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 4,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

libaio-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=9831: Wed Oct 26 15:21:52 2022
  read: IOPS=404k, BW=197MiB/s (207MB/s)(1382MiB/7001msec)
    slat (nsec): min=1653, max=22372, avg=1839.17, stdev=121.42
    clat (nsec): min=691, max=120626, avg=6717.75, stdev=2347.79
     lat (usec): min=5, max=122, avg= 8.56, stdev= 2.35
    clat percentiles (nsec):
     |  1.00th=[ 5152],  5.00th=[ 5280], 10.00th=[ 5280], 20.00th=[ 5280],
     | 30.00th=[ 5344], 40.00th=[ 5408], 50.00th=[ 7264], 60.00th=[ 7328],
     | 70.00th=[ 7328], 80.00th=[ 7392], 90.00th=[ 7584], 95.00th=[ 7968],
     | 99.00th=[17536], 99.50th=[20608], 99.90th=[26240], 99.95th=[32128],
     | 99.99th=[64256]
   bw (  KiB/s): min=201099, max=203576, per=100.00%, avg=202148.36, stdev=804.85, samples=14
   iops        : min=402198, max=407152, avg=404296.71, stdev=1609.70, samples=14
  lat (nsec)   : 750=0.01%, 1000=0.01%
  lat (usec)   : 4=0.01%, 10=96.73%, 20=2.70%, 50=0.52%, 100=0.04%
  lat (usec)   : 250=0.01%
  cpu          : usr=39.81%, sys=60.16%, ctx=12, majf=0, minf=0
  IO depths    : 1=0.1%, 2=49.2%, 4=50.8%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=2829969,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: bw=197MiB/s (207MB/s), 197MiB/s-197MiB/s (207MB/s-207MB/s), io=1382MiB (1449MB), run=7001-7001msec

$ rm /tmp/fio-output_512_4_libaio-bdev_xnvme_2.txt
rm: cannot remove '/tmp/fio-output_512_4_libaio-bdev_xnvme_2.txt': No such file or directory

$ /opt/fio/bin/fio --name="libaio-bdev_xnvme" --rw="randread" --size="12G" --iodepth="4" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_4_libaio-bdev_xnvme_2.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_512_4_libaio-bdev_xnvme_2.txt
libaio-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666790527,
  "timestamp_ms" : 1666790527394,
  "time" : "Wed Oct 26 15:22:07 2022",
  "jobs" : [
    {
      "jobname" : "libaio-bdev_xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-bdev_xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "4",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_xnvme_libaio.conf",
        "filename" : "Nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 1440870912,
        "io_kbytes" : 1407100,
        "bw_bytes" : 205809300,
        "bw" : 200985,
        "iops" : 401970.718469,
        "runtime" : 7001,
        "total_ios" : 2814197,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1673,
          "max" : 45495,
          "mean" : 1855.646163,
          "stddev" : 123.276425,
          "N" : 2814198
        },
        "clat_ns" : {
          "min" : 682,
          "max" : 118602,
          "mean" : 6813.198220,
          "stddev" : 2388.471879,
          "N" : 2814197,
          "percentile" : {
            "1.000000" : 5216,
            "5.000000" : 5280,
            "10.000000" : 5280,
            "20.000000" : 5344,
            "30.000000" : 5344,
            "40.000000" : 5472,
            "50.000000" : 7264,
            "60.000000" : 7328,
            "70.000000" : 7392,
            "80.000000" : 7456,
            "90.000000" : 7904,
            "95.000000" : 8096,
            "99.000000" : 17536,
            "99.500000" : 21376,
            "99.900000" : 26496,
            "99.950000" : 32128,
            "99.990000" : 65280
          }
        },
        "lat_ns" : {
          "min" : 5480,
          "max" : 120416,
          "mean" : 8668.844394,
          "stddev" : 2391.963514,
          "N" : 2814197
        },
        "bw_min" : 200134,
        "bw_max" : 201911,
        "bw_agg" : 100.000000,
        "bw_mean" : 201021.500000,
        "bw_dev" : 640.262897,
        "bw_samples" : 14,
        "iops_min" : 400268,
        "iops_max" : 403823,
        "iops_mean" : 402043.071429,
        "iops_stddev" : 1280.632684,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 40.957143,
      "sys_cpu" : 59.014286,
      "ctx" : 20,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.100000,
        "2" : 47.461923,
        "4" : 52.534986,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.010000,
        "1000" : 0.010000
      },
      "latency_us" : {
        "2" : 0.010000,
        "4" : 0.010000,
        "10" : 96.760248,
        "20" : 2.542715,
        "50" : 0.653970,
        "100" : 0.034681,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 4,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

libaio-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=9890: Wed Oct 26 15:22:07 2022
  read: IOPS=402k, BW=196MiB/s (206MB/s)(1374MiB/7001msec)
    slat (nsec): min=1673, max=45495, avg=1855.65, stdev=123.28
    clat (nsec): min=682, max=118602, avg=6813.20, stdev=2388.47
     lat (usec): min=5, max=120, avg= 8.67, stdev= 2.39
    clat percentiles (nsec):
     |  1.00th=[ 5216],  5.00th=[ 5280], 10.00th=[ 5280], 20.00th=[ 5344],
     | 30.00th=[ 5344], 40.00th=[ 5472], 50.00th=[ 7264], 60.00th=[ 7328],
     | 70.00th=[ 7392], 80.00th=[ 7456], 90.00th=[ 7904], 95.00th=[ 8096],
     | 99.00th=[17536], 99.50th=[21376], 99.90th=[26496], 99.95th=[32128],
     | 99.99th=[65280]
   bw (  KiB/s): min=200134, max=201911, per=100.00%, avg=201021.50, stdev=640.26, samples=14
   iops        : min=400268, max=403823, avg=402043.07, stdev=1280.63, samples=14
  lat (nsec)   : 750=0.01%, 1000=0.01%
  lat (usec)   : 2=0.01%, 4=0.01%, 10=96.76%, 20=2.54%, 50=0.65%
  lat (usec)   : 100=0.03%, 250=0.01%
  cpu          : usr=40.96%, sys=59.01%, ctx=20, majf=0, minf=0
  IO depths    : 1=0.1%, 2=47.5%, 4=52.5%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=2814197,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: bw=196MiB/s (206MB/s), 196MiB/s-196MiB/s (206MB/s-206MB/s), io=1374MiB (1441MB), run=7001-7001msec

$ rm /tmp/fio-output_512_4_libaio-xnvme_0.txt
rm: cannot remove '/tmp/fio-output_512_4_libaio-xnvme_0.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring-reference" --rw="randread" --size="12G" --iodepth="1" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_1_io_uring-reference_0.txt" --cpus_allowed="1"
$ /opt/fio/bin/fio --name="libaio-xnvme" --rw="randread" --size="12G" --iodepth="4" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="libaio" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_4_libaio-xnvme_0.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_4_libaio-xnvme_0.txt
libaio-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666790541,
  "timestamp_ms" : 1666790541496,
  "time" : "Wed Oct 26 15:22:21 2022",
  "jobs" : [
    {
      "jobname" : "libaio-xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "4",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
        "xnvme_be" : "linux",
        "xnvme_async" : "libaio",
        "xnvme_sync" : "nvme",
        "xnvme_admin" : "nvme",
        "xnvme_dev_nsid" : "1",
        "filename" : "/dev/nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 1465430528,
        "io_kbytes" : 1431084,
        "bw_bytes" : 209317315,
        "bw" : 204411,
        "iops" : 408822.453935,
        "runtime" : 7001,
        "total_ios" : 2862166,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1502,
          "max" : 21721,
          "mean" : 1667.778486,
          "stddev" : 106.823115,
          "N" : 2862166
        },
        "clat_ns" : {
          "min" : 511,
          "max" : 149921,
          "mean" : 7891.621895,
          "stddev" : 2696.853545,
          "N" : 2862169,
          "percentile" : {
            "1.000000" : 7264,
            "5.000000" : 7328,
            "10.000000" : 7328,
            "20.000000" : 7328,
            "30.000000" : 7392,
            "40.000000" : 7392,
            "50.000000" : 7392,
            "60.000000" : 7456,
            "70.000000" : 7520,
            "80.000000" : 7648,
            "90.000000" : 7776,
            "95.000000" : 7840,
            "99.000000" : 22656,
            "99.500000" : 27264,
            "99.900000" : 33536,
            "99.950000" : 38656,
            "99.990000" : 72192
          }
        },
        "lat_ns" : {
          "min" : 6402,
          "max" : 151595,
          "mean" : 9559.400362,
          "stddev" : 2700.206986,
          "N" : 2862169
        },
        "bw_min" : 203864,
        "bw_max" : 205193,
        "bw_agg" : 100.000000,
        "bw_mean" : 204511.357143,
        "bw_dev" : 307.759798,
        "bw_samples" : 14,
        "iops_min" : 407728,
        "iops_max" : 410386,
        "iops_mean" : 409022.714286,
        "iops_stddev" : 615.561711,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 27.228571,
      "sys_cpu" : 69.314286,
      "ctx" : 25260,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.010000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 96.581889,
        "20" : 2.139429,
        "50" : 1.239236,
        "100" : 0.038922,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 4,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

libaio-xnvme: (groupid=0, jobs=1): err= 0: pid=9937: Wed Oct 26 15:22:21 2022
  read: IOPS=409k, BW=200MiB/s (209MB/s)(1398MiB/7001msec)
    slat (nsec): min=1502, max=21721, avg=1667.78, stdev=106.82
    clat (nsec): min=511, max=149921, avg=7891.62, stdev=2696.85
     lat (usec): min=6, max=151, avg= 9.56, stdev= 2.70
    clat percentiles (nsec):
     |  1.00th=[ 7264],  5.00th=[ 7328], 10.00th=[ 7328], 20.00th=[ 7328],
     | 30.00th=[ 7392], 40.00th=[ 7392], 50.00th=[ 7392], 60.00th=[ 7456],
     | 70.00th=[ 7520], 80.00th=[ 7648], 90.00th=[ 7776], 95.00th=[ 7840],
     | 99.00th=[22656], 99.50th=[27264], 99.90th=[33536], 99.95th=[38656],
     | 99.99th=[72192]
   bw (  KiB/s): min=203864, max=205193, per=100.00%, avg=204511.36, stdev=307.76, samples=14
   iops        : min=407728, max=410386, avg=409022.71, stdev=615.56, samples=14
  lat (nsec)   : 750=0.01%
  lat (usec)   : 10=96.58%, 20=2.14%, 50=1.24%, 100=0.04%, 250=0.01%
  cpu          : usr=27.23%, sys=69.31%, ctx=25260, majf=0, minf=0
  IO depths    : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=2862166,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: bw=200MiB/s (209MB/s), 200MiB/s-200MiB/s (209MB/s-209MB/s), io=1398MiB (1465MB), run=7001-7001msec

$ rm /tmp/fio-output_512_4_libaio-xnvme_1.txt
rm: cannot remove '/tmp/fio-output_512_4_libaio-xnvme_1.txt': No such file or directory

$ /opt/fio/bin/fio --name="libaio-xnvme" --rw="randread" --size="12G" --iodepth="4" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="libaio" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_4_libaio-xnvme_1.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_4_libaio-xnvme_1.txt
libaio-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666790555,
  "timestamp_ms" : 1666790555276,
  "time" : "Wed Oct 26 15:22:35 2022",
  "jobs" : [
    {
      "jobname" : "libaio-xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "4",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
        "xnvme_be" : "linux",
        "xnvme_async" : "libaio",
        "xnvme_sync" : "nvme",
        "xnvme_admin" : "nvme",
        "xnvme_dev_nsid" : "1",
        "filename" : "/dev/nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 1449910272,
        "io_kbytes" : 1415928,
        "bw_bytes" : 207100453,
        "bw" : 202246,
        "iops" : 404492.501071,
        "runtime" : 7001,
        "total_ios" : 2831852,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1533,
          "max" : 21871,
          "mean" : 1679.587689,
          "stddev" : 109.338587,
          "N" : 2831853
        },
        "clat_ns" : {
          "min" : 4919,
          "max" : 90730,
          "mean" : 7972.974219,
          "stddev" : 2662.447268,
          "N" : 2831855,
          "percentile" : {
            "1.000000" : 7392,
            "5.000000" : 7392,
            "10.000000" : 7456,
            "20.000000" : 7456,
            "30.000000" : 7456,
            "40.000000" : 7520,
            "50.000000" : 7520,
            "60.000000" : 7520,
            "70.000000" : 7584,
            "80.000000" : 7648,
            "90.000000" : 7840,
            "95.000000" : 7968,
            "99.000000" : 22400,
            "99.500000" : 27264,
            "99.900000" : 33024,
            "99.950000" : 36608,
            "99.990000" : 71168
          }
        },
        "lat_ns" : {
          "min" : 6472,
          "max" : 92383,
          "mean" : 9652.561871,
          "stddev" : 2665.381488,
          "N" : 2831855
        },
        "bw_min" : 201817,
        "bw_max" : 203505,
        "bw_agg" : 100.000000,
        "bw_mean" : 202362.923077,
        "bw_dev" : 461.921072,
        "bw_samples" : 13,
        "iops_min" : 403634,
        "iops_max" : 407010,
        "iops_mean" : 404726.153846,
        "iops_stddev" : 923.775843,
        "iops_samples" : 13
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 32.457143,
      "sys_cpu" : 64.142857,
      "ctx" : 25084,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 96.526902,
        "20" : 2.143403,
        "50" : 1.292864,
        "100" : 0.036937,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 4,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

libaio-xnvme: (groupid=0, jobs=1): err= 0: pid=9985: Wed Oct 26 15:22:35 2022
  read: IOPS=404k, BW=198MiB/s (207MB/s)(1383MiB/7001msec)
    slat (nsec): min=1533, max=21871, avg=1679.59, stdev=109.34
    clat (nsec): min=4919, max=90730, avg=7972.97, stdev=2662.45
     lat (nsec): min=6472, max=92383, avg=9652.56, stdev=2665.38
    clat percentiles (nsec):
     |  1.00th=[ 7392],  5.00th=[ 7392], 10.00th=[ 7456], 20.00th=[ 7456],
     | 30.00th=[ 7456], 40.00th=[ 7520], 50.00th=[ 7520], 60.00th=[ 7520],
     | 70.00th=[ 7584], 80.00th=[ 7648], 90.00th=[ 7840], 95.00th=[ 7968],
     | 99.00th=[22400], 99.50th=[27264], 99.90th=[33024], 99.95th=[36608],
     | 99.99th=[71168]
   bw (  KiB/s): min=201817, max=203505, per=100.00%, avg=202362.92, stdev=461.92, samples=13
   iops        : min=403634, max=407010, avg=404726.15, stdev=923.78, samples=13
  lat (usec)   : 10=96.53%, 20=2.14%, 50=1.29%, 100=0.04%
  cpu          : usr=32.46%, sys=64.14%, ctx=25084, majf=0, minf=0
  IO depths    : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=2831852,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: bw=198MiB/s (207MB/s), 198MiB/s-198MiB/s (207MB/s-207MB/s), io=1383MiB (1450MB), run=7001-7001msec

$ rm /tmp/fio-output_512_4_libaio-xnvme_2.txt
rm: cannot remove '/tmp/fio-output_512_4_libaio-xnvme_2.txt': No such file or directory

$ /opt/fio/bin/fio --name="libaio-xnvme" --rw="randread" --size="12G" --iodepth="4" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="libaio" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_4_libaio-xnvme_2.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_4_libaio-xnvme_2.txt
libaio-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666790570,
  "timestamp_ms" : 1666790570033,
  "time" : "Wed Oct 26 15:22:50 2022",
  "jobs" : [
    {
      "jobname" : "libaio-xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "4",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
        "xnvme_be" : "linux",
        "xnvme_async" : "libaio",
        "xnvme_sync" : "nvme",
        "xnvme_admin" : "nvme",
        "xnvme_dev_nsid" : "1",
        "filename" : "/dev/nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 1468516864,
        "io_kbytes" : 1434098,
        "bw_bytes" : 209758157,
        "bw" : 204841,
        "iops" : 409683.330953,
        "runtime" : 7001,
        "total_ios" : 2868193,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1513,
          "max" : 21661,
          "mean" : 1664.947021,
          "stddev" : 111.275646,
          "N" : 2868194
        },
        "clat_ns" : {
          "min" : 4869,
          "max" : 125125,
          "mean" : 7874.002652,
          "stddev" : 2616.785369,
          "N" : 2868196,
          "percentile" : {
            "1.000000" : 7264,
            "5.000000" : 7328,
            "10.000000" : 7328,
            "20.000000" : 7392,
            "30.000000" : 7392,
            "40.000000" : 7392,
            "50.000000" : 7392,
            "60.000000" : 7456,
            "70.000000" : 7456,
            "80.000000" : 7520,
            "90.000000" : 7712,
            "95.000000" : 7840,
            "99.000000" : 21888,
            "99.500000" : 25984,
            "99.900000" : 33024,
            "99.950000" : 36608,
            "99.990000" : 70144
          }
        },
        "lat_ns" : {
          "min" : 6412,
          "max" : 126818,
          "mean" : 9538.949647,
          "stddev" : 2618.618149,
          "N" : 2868196
        },
        "bw_min" : 203662,
        "bw_max" : 205924,
        "bw_agg" : 100.000000,
        "bw_mean" : 204950.642857,
        "bw_dev" : 680.498189,
        "bw_samples" : 14,
        "iops_min" : 407324,
        "iops_max" : 411848,
        "iops_mean" : 409901.571429,
        "iops_stddev" : 1360.938913,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 32.400000,
      "sys_cpu" : 64.171429,
      "ctx" : 25312,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 96.547025,
        "20" : 2.215506,
        "50" : 1.198664,
        "100" : 0.038526,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 4,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

libaio-xnvme: (groupid=0, jobs=1): err= 0: pid=10034: Wed Oct 26 15:22:50 2022
  read: IOPS=410k, BW=200MiB/s (210MB/s)(1400MiB/7001msec)
    slat (nsec): min=1513, max=21661, avg=1664.95, stdev=111.28
    clat (usec): min=4, max=125, avg= 7.87, stdev= 2.62
     lat (usec): min=6, max=126, avg= 9.54, stdev= 2.62
    clat percentiles (nsec):
     |  1.00th=[ 7264],  5.00th=[ 7328], 10.00th=[ 7328], 20.00th=[ 7392],
     | 30.00th=[ 7392], 40.00th=[ 7392], 50.00th=[ 7392], 60.00th=[ 7456],
     | 70.00th=[ 7456], 80.00th=[ 7520], 90.00th=[ 7712], 95.00th=[ 7840],
     | 99.00th=[21888], 99.50th=[25984], 99.90th=[33024], 99.95th=[36608],
     | 99.99th=[70144]
   bw (  KiB/s): min=203662, max=205924, per=100.00%, avg=204950.64, stdev=680.50, samples=14
   iops        : min=407324, max=411848, avg=409901.57, stdev=1360.94, samples=14
  lat (usec)   : 10=96.55%, 20=2.22%, 50=1.20%, 100=0.04%, 250=0.01%
  cpu          : usr=32.40%, sys=64.17%, ctx=25312, majf=0, minf=0
  IO depths    : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=2868193,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: bw=200MiB/s (210MB/s), 200MiB/s-200MiB/s (210MB/s-210MB/s), io=1400MiB (1469MB), run=7001-7001msec

$ rm /tmp/fio-output_512_8_io_uring_cmd-reference_0.txt
rm: cannot remove '/tmp/fio-output_512_8_io_uring_cmd-reference_0.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring_cmd-reference" --rw="randread" --size="12G" --iodepth="8" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring_cmd" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_512_8_io_uring_cmd-reference_0.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_1_io_uring-reference_0.txt
io_uring-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=io_uring, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666789277,
  "timestamp_ms" : 1666789277197,
  "time" : "Wed Oct 26 15:01:17 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-reference",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-reference",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "1",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "io_uring",
        "filename" : "/dev/nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 392608768,
        "io_kbytes" : 383407,
        "bw_bytes" : 56078955,
        "bw" : 54764,
        "iops" : 109529.067276,
        "runtime" : 7001,
        "total_ios" : 766813,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1443,
          "max" : 17152,
          "mean" : 1576.314540,
          "stddev" : 85.440023,
          "N" : 766814
        },
        "clat_ns" : {
          "min" : 161,
          "max" : 78006,
          "mean" : 7235.024115,
          "stddev" : 971.994924,
          "N" : 766813,
          "percentile" : {
            "1.000000" : 6752,
            "5.000000" : 6944,
            "10.000000" : 7008,
            "20.000000" : 7008,
            "30.000000" : 7072,
            "40.000000" : 7072,
            "50.000000" : 7136,
            "60.000000" : 7200,
            "70.000000" : 7200,
            "80.000000" : 7264,
            "90.000000" : 7328,
            "95.000000" : 7456,
            "99.000000" : 10304,
            "99.500000" : 13504,
            "99.900000" : 15552,
            "99.950000" : 15808,
            "99.990000" : 58112
          }
        },
        "lat_ns" : {
          "min" : 7444,
          "max" : 79509,
          "mean" : 8811.338725,
          "stddev" : 977.342601,
          "N" : 766813
        },
        "bw_min" : 54549,
        "bw_max" : 55065,
        "bw_agg" : 100.000000,
        "bw_mean" : 54801.642857,
        "bw_dev" : 147.506249,
        "bw_samples" : 14,
        "iops_min" : 109098,
        "iops_max" : 110130,
        "iops_mean" : 109603.500000,
        "iops_stddev" : 295.184883,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 9.400000,
      "sys_cpu" : 34.500000,
      "ctx" : 766814,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 100.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010000,
        "10" : 98.745718,
        "20" : 1.234330,
        "50" : 0.010000,
        "100" : 0.013954,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 1,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ],
  "disk_util" : [
    {
      "name" : "nvme0n1",
      "read_ios" : 1099741,
      "write_ios" : 0,
      "read_merges" : 0,
      "write_merges" : 0,
      "read_ticks" : 5523,
      "write_ticks" : 0,
      "in_queue" : 5523,
      "util" : 99.108381
    }
  ]
}

io_uring-reference: (groupid=0, jobs=1): err= 0: pid=5391: Wed Oct 26 15:01:17 2022
  read: IOPS=110k, BW=53.5MiB/s (56.1MB/s)(374MiB/7001msec)
    slat (nsec): min=1443, max=17152, avg=1576.31, stdev=85.44
    clat (nsec): min=161, max=78006, avg=7235.02, stdev=971.99
     lat (nsec): min=7444, max=79509, avg=8811.34, stdev=977.34
    clat percentiles (nsec):
     |  1.00th=[ 6752],  5.00th=[ 6944], 10.00th=[ 7008], 20.00th=[ 7008],
     | 30.00th=[ 7072], 40.00th=[ 7072], 50.00th=[ 7136], 60.00th=[ 7200],
     | 70.00th=[ 7200], 80.00th=[ 7264], 90.00th=[ 7328], 95.00th=[ 7456],
     | 99.00th=[10304], 99.50th=[13504], 99.90th=[15552], 99.95th=[15808],
     | 99.99th=[58112]
   bw (  KiB/s): min=54549, max=55065, per=100.00%, avg=54801.64, stdev=147.51, samples=14
   iops        : min=109098, max=110130, avg=109603.50, stdev=295.18, samples=14
  lat (nsec)   : 250=0.01%
  lat (usec)   : 4=0.01%, 10=98.75%, 20=1.23%, 50=0.01%, 100=0.01%
  cpu          : usr=9.40%, sys=34.50%, ctx=766814, majf=0, minf=0
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=766813,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=53.5MiB/s (56.1MB/s), 53.5MiB/s-53.5MiB/s (56.1MB/s-56.1MB/s), io=374MiB (393MB), run=7001-7001msec

Disk stats (read/write):
  nvme0n1: ios=1099741/0, merge=0/0, ticks=5523/0, in_queue=5523, util=99.11%

$ cat /tmp/fio-output_512_8_io_uring_cmd-reference_0.txt
io_uring_cmd-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=io_uring_cmd, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666790583,
  "timestamp_ms" : 1666790583831,
  "time" : "Wed Oct 26 15:23:03 2022",
  "jobs" : [
    {
      "jobname" : "io_uring_cmd-reference",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring_cmd-reference",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "8",
        "bs" : "512",
        "direct" : "0",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "io_uring_cmd",
        "filename" : "/dev/ng0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 1613466624,
        "io_kbytes" : 1575651,
        "bw_bytes" : 230462308,
        "bw" : 225060,
        "iops" : 450120.554207,
        "runtime" : 7001,
        "total_ios" : 3151294,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1172,
          "max" : 51907,
          "mean" : 1784.391308,
          "stddev" : 386.919507,
          "N" : 3151295
        },
        "clat_ns" : {
          "min" : 5240,
          "max" : 113272,
          "mean" : 15718.233979,
          "stddev" : 3820.322070,
          "N" : 3151301,
          "percentile" : {
            "1.000000" : 12224,
            "5.000000" : 12736,
            "10.000000" : 13120,
            "20.000000" : 14272,
            "30.000000" : 14784,
            "40.000000" : 14912,
            "50.000000" : 15040,
            "60.000000" : 15424,
            "70.000000" : 15680,
            "80.000000" : 16320,
            "90.000000" : 17280,
            "95.000000" : 18048,
            "99.000000" : 36608,
            "99.500000" : 39168,
            "99.900000" : 48384,
            "99.950000" : 64256,
            "99.990000" : 79360
          }
        },
        "lat_ns" : {
          "min" : 6453,
          "max" : 114505,
          "mean" : 17502.625444,
          "stddev" : 3781.638897,
          "N" : 3151301
        },
        "bw_min" : 224723,
        "bw_max" : 225795,
        "bw_agg" : 100.000000,
        "bw_mean" : 225187.928571,
        "bw_dev" : 301.062085,
        "bw_samples" : 14,
        "iops_min" : 449446,
        "iops_max" : 451591,
        "iops_mean" : 450376.071429,
        "iops_stddev" : 602.094097,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 16.871429,
      "sys_cpu" : 79.900000,
      "ctx" : 15021,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 100.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 99.999968,
        "8" : 0.100000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.044109,
        "20" : 96.321670,
        "50" : 3.541847,
        "100" : 0.092248,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 8,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring_cmd-reference: (groupid=0, jobs=1): err= 0: pid=10082: Wed Oct 26 15:23:03 2022
  read: IOPS=450k, BW=220MiB/s (230MB/s)(1539MiB/7001msec)
    slat (nsec): min=1172, max=51907, avg=1784.39, stdev=386.92
    clat (usec): min=5, max=113, avg=15.72, stdev= 3.82
     lat (usec): min=6, max=114, avg=17.50, stdev= 3.78
    clat percentiles (nsec):
     |  1.00th=[12224],  5.00th=[12736], 10.00th=[13120], 20.00th=[14272],
     | 30.00th=[14784], 40.00th=[14912], 50.00th=[15040], 60.00th=[15424],
     | 70.00th=[15680], 80.00th=[16320], 90.00th=[17280], 95.00th=[18048],
     | 99.00th=[36608], 99.50th=[39168], 99.90th=[48384], 99.95th=[64256],
     | 99.99th=[79360]
   bw (  KiB/s): min=224723, max=225795, per=100.00%, avg=225187.93, stdev=301.06, samples=14
   iops        : min=449446, max=451591, avg=450376.07, stdev=602.09, samples=14
  lat (usec)   : 10=0.04%, 20=96.32%, 50=3.54%, 100=0.09%, 250=0.01%
  cpu          : usr=16.87%, sys=79.90%, ctx=15021, majf=0, minf=0
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=3151294,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
   READ: bw=220MiB/s (230MB/s), 220MiB/s-220MiB/s (230MB/s-230MB/s), io=1539MiB (1613MB), run=7001-7001msec

$ rm /tmp/fio-output_512_8_io_uring_cmd-reference_1.txt
rm: cannot remove '/tmp/fio-output_512_8_io_uring_cmd-reference_1.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring_cmd-reference" --rw="randread" --size="12G" --iodepth="8" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring_cmd" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_512_8_io_uring_cmd-reference_1.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_8_io_uring_cmd-reference_1.txt
io_uring_cmd-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=io_uring_cmd, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666790597,
  "timestamp_ms" : 1666790597635,
  "time" : "Wed Oct 26 15:23:17 2022",
  "jobs" : [
    {
      "jobname" : "io_uring_cmd-reference",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring_cmd-reference",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "8",
        "bs" : "512",
        "direct" : "0",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "io_uring_cmd",
        "filename" : "/dev/ng0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 1596023808,
        "io_kbytes" : 1558617,
        "bw_bytes" : 227970833,
        "bw" : 222627,
        "iops" : 445254.392230,
        "runtime" : 7001,
        "total_ios" : 3117226,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1192,
          "max" : 26660,
          "mean" : 1791.133760,
          "stddev" : 379.567079,
          "N" : 3117227
        },
        "clat_ns" : {
          "min" : 1963,
          "max" : 125385,
          "mean" : 15887.595758,
          "stddev" : 3772.244360,
          "N" : 3117233,
          "percentile" : {
            "1.000000" : 12224,
            "5.000000" : 12864,
            "10.000000" : 13248,
            "20.000000" : 14528,
            "30.000000" : 14912,
            "40.000000" : 15168,
            "50.000000" : 15296,
            "60.000000" : 15680,
            "70.000000" : 15936,
            "80.000000" : 16512,
            "90.000000" : 17536,
            "95.000000" : 18048,
            "99.000000" : 36608,
            "99.500000" : 38656,
            "99.900000" : 47872,
            "99.950000" : 63744,
            "99.990000" : 79360
          }
        },
        "lat_ns" : {
          "min" : 6492,
          "max" : 126838,
          "mean" : 17678.729535,
          "stddev" : 3741.390408,
          "N" : 3117233
        },
        "bw_min" : 221913,
        "bw_max" : 223653,
        "bw_agg" : 100.000000,
        "bw_mean" : 222667.071429,
        "bw_dev" : 489.091224,
        "bw_samples" : 14,
        "iops_min" : 443826,
        "iops_max" : 447306,
        "iops_mean" : 445334.142857,
        "iops_stddev" : 978.182448,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 17.685714,
      "sys_cpu" : 79.200000,
      "ctx" : 14928,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 100.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 99.999968,
        "8" : 0.100000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.010000,
        "4" : 0.010000,
        "10" : 0.050077,
        "20" : 96.374341,
        "50" : 3.485727,
        "100" : 0.089374,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 8,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring_cmd-reference: (groupid=0, jobs=1): err= 0: pid=10128: Wed Oct 26 15:23:17 2022
  read: IOPS=445k, BW=217MiB/s (228MB/s)(1522MiB/7001msec)
    slat (nsec): min=1192, max=26660, avg=1791.13, stdev=379.57
    clat (nsec): min=1963, max=125385, avg=15887.60, stdev=3772.24
     lat (usec): min=6, max=126, avg=17.68, stdev= 3.74
    clat percentiles (nsec):
     |  1.00th=[12224],  5.00th=[12864], 10.00th=[13248], 20.00th=[14528],
     | 30.00th=[14912], 40.00th=[15168], 50.00th=[15296], 60.00th=[15680],
     | 70.00th=[15936], 80.00th=[16512], 90.00th=[17536], 95.00th=[18048],
     | 99.00th=[36608], 99.50th=[38656], 99.90th=[47872], 99.95th=[63744],
     | 99.99th=[79360]
   bw (  KiB/s): min=221913, max=223653, per=100.00%, avg=222667.07, stdev=489.09, samples=14
   iops        : min=443826, max=447306, avg=445334.14, stdev=978.18, samples=14
  lat (usec)   : 2=0.01%, 4=0.01%, 10=0.05%, 20=96.37%, 50=3.49%
  lat (usec)   : 100=0.09%, 250=0.01%
  cpu          : usr=17.69%, sys=79.20%, ctx=14928, majf=0, minf=0
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=3117226,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
   READ: bw=217MiB/s (228MB/s), 217MiB/s-217MiB/s (228MB/s-228MB/s), io=1522MiB (1596MB), run=7001-7001msec

$ rm /tmp/fio-output_512_8_io_uring_cmd-reference_2.txt
rm: cannot remove '/tmp/fio-output_512_8_io_uring_cmd-reference_2.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring_cmd-reference" --rw="randread" --size="12G" --iodepth="8" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring_cmd" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_512_8_io_uring_cmd-reference_2.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_8_io_uring_cmd-reference_2.txt
io_uring_cmd-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=io_uring_cmd, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666790611,
  "timestamp_ms" : 1666790611370,
  "time" : "Wed Oct 26 15:23:31 2022",
  "jobs" : [
    {
      "jobname" : "io_uring_cmd-reference",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring_cmd-reference",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "8",
        "bs" : "512",
        "direct" : "0",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "io_uring_cmd",
        "filename" : "/dev/ng0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 1604506112,
        "io_kbytes" : 1566900,
        "bw_bytes" : 229182418,
        "bw" : 223810,
        "iops" : 447620.768462,
        "runtime" : 7001,
        "total_ios" : 3133793,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1162,
          "max" : 23243,
          "mean" : 1790.342289,
          "stddev" : 400.121691,
          "N" : 3133794
        },
        "clat_ns" : {
          "min" : 1874,
          "max" : 130384,
          "mean" : 15806.641568,
          "stddev" : 3805.133567,
          "N" : 3133800,
          "percentile" : {
            "1.000000" : 12096,
            "5.000000" : 12736,
            "10.000000" : 13248,
            "20.000000" : 14400,
            "30.000000" : 14784,
            "40.000000" : 15040,
            "50.000000" : 15168,
            "60.000000" : 15552,
            "70.000000" : 15808,
            "80.000000" : 16512,
            "90.000000" : 17280,
            "95.000000" : 18048,
            "99.000000" : 36608,
            "99.500000" : 38656,
            "99.900000" : 48384,
            "99.950000" : 64256,
            "99.990000" : 79360
          }
        },
        "lat_ns" : {
          "min" : 6532,
          "max" : 131657,
          "mean" : 17596.983862,
          "stddev" : 3768.535898,
          "N" : 3133800
        },
        "bw_min" : 222819,
        "bw_max" : 225676,
        "bw_agg" : 100.000000,
        "bw_mean" : 223915.384615,
        "bw_dev" : 891.737867,
        "bw_samples" : 13,
        "iops_min" : 445638,
        "iops_max" : 451352,
        "iops_mean" : 447830.846154,
        "iops_stddev" : 1783.386285,
        "iops_samples" : 13
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 17.214286,
      "sys_cpu" : 79.585714,
      "ctx" : 14974,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 100.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 99.999968,
        "8" : 0.100000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.010000,
        "4" : 0.000000,
        "10" : 0.049142,
        "20" : 96.312360,
        "50" : 3.546022,
        "100" : 0.092412,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 8,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring_cmd-reference: (groupid=0, jobs=1): err= 0: pid=10175: Wed Oct 26 15:23:31 2022
  read: IOPS=448k, BW=219MiB/s (229MB/s)(1530MiB/7001msec)
    slat (nsec): min=1162, max=23243, avg=1790.34, stdev=400.12
    clat (nsec): min=1874, max=130384, avg=15806.64, stdev=3805.13
     lat (usec): min=6, max=131, avg=17.60, stdev= 3.77
    clat percentiles (nsec):
     |  1.00th=[12096],  5.00th=[12736], 10.00th=[13248], 20.00th=[14400],
     | 30.00th=[14784], 40.00th=[15040], 50.00th=[15168], 60.00th=[15552],
     | 70.00th=[15808], 80.00th=[16512], 90.00th=[17280], 95.00th=[18048],
     | 99.00th=[36608], 99.50th=[38656], 99.90th=[48384], 99.95th=[64256],
     | 99.99th=[79360]
   bw (  KiB/s): min=222819, max=225676, per=100.00%, avg=223915.38, stdev=891.74, samples=13
   iops        : min=445638, max=451352, avg=447830.85, stdev=1783.39, samples=13
  lat (usec)   : 2=0.01%, 10=0.05%, 20=96.31%, 50=3.55%, 100=0.09%
  lat (usec)   : 250=0.01%
  cpu          : usr=17.21%, sys=79.59%, ctx=14974, majf=0, minf=0
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=3133793,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
   READ: bw=219MiB/s (229MB/s), 219MiB/s-219MiB/s (229MB/s-229MB/s), io=1530MiB (1605MB), run=7001-7001msec

$ rm /tmp/fio-output_512_8_io_uring_cmd-xnvme_0.txt
rm: cannot remove '/tmp/fio-output_512_8_io_uring_cmd-xnvme_0.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring_cmd-xnvme" --rw="randread" --size="12G" --iodepth="8" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring_cmd" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_512_8_io_uring_cmd-xnvme_0.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_8_io_uring_cmd-xnvme_0.txt
io_uring_cmd-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666790625,
  "timestamp_ms" : 1666790625309,
  "time" : "Wed Oct 26 15:23:45 2022",
  "jobs" : [
    {
      "jobname" : "io_uring_cmd-xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring_cmd-xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "8",
        "bs" : "512",
        "direct" : "0",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
        "xnvme_be" : "linux",
        "xnvme_async" : "io_uring_cmd",
        "xnvme_sync" : "nvme",
        "xnvme_admin" : "nvme",
        "xnvme_dev_nsid" : "1",
        "filename" : "/dev/ng0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 1514349056,
        "io_kbytes" : 1478856,
        "bw_bytes" : 216335579,
        "bw" : 211265,
        "iops" : 422529.285714,
        "runtime" : 7000,
        "total_ios" : 2957705,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1232,
          "max" : 23514,
          "mean" : 1841.139255,
          "stddev" : 389.694801,
          "N" : 2957706
        },
        "clat_ns" : {
          "min" : 3677,
          "max" : 120536,
          "mean" : 16780.929818,
          "stddev" : 3696.313221,
          "N" : 2957712,
          "percentile" : {
            "1.000000" : 12608,
            "5.000000" : 13248,
            "10.000000" : 14400,
            "20.000000" : 15296,
            "30.000000" : 15808,
            "40.000000" : 16192,
            "50.000000" : 16512,
            "60.000000" : 16768,
            "70.000000" : 17024,
            "80.000000" : 17536,
            "90.000000" : 17792,
            "95.000000" : 18560,
            "99.000000" : 36608,
            "99.500000" : 38656,
            "99.900000" : 50432,
            "99.950000" : 66048,
            "99.990000" : 81408
          }
        },
        "lat_ns" : {
          "min" : 6672,
          "max" : 121879,
          "mean" : 18622.069282,
          "stddev" : 3671.329577,
          "N" : 2957712
        },
        "bw_min" : 208536,
        "bw_max" : 213329,
        "bw_agg" : 100.000000,
        "bw_mean" : 211349.428571,
        "bw_dev" : 1221.796830,
        "bw_samples" : 14,
        "iops_min" : 417072,
        "iops_max" : 426658,
        "iops_mean" : 422699.000000,
        "iops_stddev" : 2443.594893,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 20.442857,
      "sys_cpu" : 76.714286,
      "ctx" : 14856,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 100.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 99.999983,
        "8" : 0.100000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010000,
        "10" : 0.040910,
        "20" : 96.235764,
        "50" : 3.622944,
        "100" : 0.100111,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 8,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring_cmd-xnvme: (groupid=0, jobs=1): err= 0: pid=10221: Wed Oct 26 15:23:45 2022
  read: IOPS=423k, BW=206MiB/s (216MB/s)(1444MiB/7000msec)
    slat (nsec): min=1232, max=23514, avg=1841.14, stdev=389.69
    clat (usec): min=3, max=120, avg=16.78, stdev= 3.70
     lat (usec): min=6, max=121, avg=18.62, stdev= 3.67
    clat percentiles (nsec):
     |  1.00th=[12608],  5.00th=[13248], 10.00th=[14400], 20.00th=[15296],
     | 30.00th=[15808], 40.00th=[16192], 50.00th=[16512], 60.00th=[16768],
     | 70.00th=[17024], 80.00th=[17536], 90.00th=[17792], 95.00th=[18560],
     | 99.00th=[36608], 99.50th=[38656], 99.90th=[50432], 99.95th=[66048],
     | 99.99th=[81408]
   bw (  KiB/s): min=208536, max=213329, per=100.00%, avg=211349.43, stdev=1221.80, samples=14
   iops        : min=417072, max=426658, avg=422699.00, stdev=2443.59, samples=14
  lat (usec)   : 4=0.01%, 10=0.04%, 20=96.24%, 50=3.62%, 100=0.10%
  lat (usec)   : 250=0.01%
  cpu          : usr=20.44%, sys=76.71%, ctx=14856, majf=0, minf=0
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=2957705,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
   READ: bw=206MiB/s (216MB/s), 206MiB/s-206MiB/s (216MB/s-216MB/s), io=1444MiB (1514MB), run=7000-7000msec

$ rm /tmp/fio-output_512_1_io_uring-reference_1.txt
rm: cannot remove '/tmp/fio-output_512_1_io_uring-reference_1.txt': No such file or directory

$ rm /tmp/fio-output_512_8_io_uring_cmd-xnvme_1.txt
rm: cannot remove '/tmp/fio-output_512_8_io_uring_cmd-xnvme_1.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring_cmd-xnvme" --rw="randread" --size="12G" --iodepth="8" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring_cmd" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_512_8_io_uring_cmd-xnvme_1.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_8_io_uring_cmd-xnvme_1.txt
io_uring_cmd-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666790639,
  "timestamp_ms" : 1666790639058,
  "time" : "Wed Oct 26 15:23:59 2022",
  "jobs" : [
    {
      "jobname" : "io_uring_cmd-xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring_cmd-xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "8",
        "bs" : "512",
        "direct" : "0",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
        "xnvme_be" : "linux",
        "xnvme_async" : "io_uring_cmd",
        "xnvme_sync" : "nvme",
        "xnvme_admin" : "nvme",
        "xnvme_dev_nsid" : "1",
        "filename" : "/dev/ng0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 1536173056,
        "io_kbytes" : 1500169,
        "bw_bytes" : 219421947,
        "bw" : 214279,
        "iops" : 428557.491787,
        "runtime" : 7001,
        "total_ios" : 3000331,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1222,
          "max" : 23965,
          "mean" : 1803.258623,
          "stddev" : 390.548360,
          "N" : 3000331
        },
        "clat_ns" : {
          "min" : 4839,
          "max" : 128591,
          "mean" : 16544.098031,
          "stddev" : 3761.336130,
          "N" : 3000338,
          "percentile" : {
            "1.000000" : 12608,
            "5.000000" : 13120,
            "10.000000" : 13760,
            "20.000000" : 15040,
            "30.000000" : 15552,
            "40.000000" : 15936,
            "50.000000" : 16192,
            "60.000000" : 16512,
            "70.000000" : 16768,
            "80.000000" : 17280,
            "90.000000" : 17536,
            "95.000000" : 18560,
            "99.000000" : 37120,
            "99.500000" : 39168,
            "99.900000" : 51456,
            "99.950000" : 67072,
            "99.990000" : 81408
          }
        },
        "lat_ns" : {
          "min" : 6191,
          "max" : 129934,
          "mean" : 18347.356140,
          "stddev" : 3737.823304,
          "N" : 3000338
        },
        "bw_min" : 211728,
        "bw_max" : 216886,
        "bw_agg" : 100.000000,
        "bw_mean" : 214318.285714,
        "bw_dev" : 1642.356301,
        "bw_samples" : 14,
        "iops_min" : 423456,
        "iops_max" : 433772,
        "iops_mean" : 428636.857143,
        "iops_stddev" : 3284.857914,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 22.057143,
      "sys_cpu" : 75.014286,
      "ctx" : 14798,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 100.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 99.999983,
        "8" : 0.100000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.043895,
        "20" : 96.254080,
        "50" : 3.599036,
        "100" : 0.101522,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 8,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring_cmd-xnvme: (groupid=0, jobs=1): err= 0: pid=10267: Wed Oct 26 15:23:59 2022
  read: IOPS=429k, BW=209MiB/s (219MB/s)(1465MiB/7001msec)
    slat (nsec): min=1222, max=23965, avg=1803.26, stdev=390.55
    clat (usec): min=4, max=128, avg=16.54, stdev= 3.76
     lat (usec): min=6, max=129, avg=18.35, stdev= 3.74
    clat percentiles (nsec):
     |  1.00th=[12608],  5.00th=[13120], 10.00th=[13760], 20.00th=[15040],
     | 30.00th=[15552], 40.00th=[15936], 50.00th=[16192], 60.00th=[16512],
     | 70.00th=[16768], 80.00th=[17280], 90.00th=[17536], 95.00th=[18560],
     | 99.00th=[37120], 99.50th=[39168], 99.90th=[51456], 99.95th=[67072],
     | 99.99th=[81408]
   bw (  KiB/s): min=211728, max=216886, per=100.00%, avg=214318.29, stdev=1642.36, samples=14
   iops        : min=423456, max=433772, avg=428636.86, stdev=3284.86, samples=14
  lat (usec)   : 10=0.04%, 20=96.25%, 50=3.60%, 100=0.10%, 250=0.01%
  cpu          : usr=22.06%, sys=75.01%, ctx=14798, majf=0, minf=0
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=3000331,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
   READ: bw=209MiB/s (219MB/s), 209MiB/s-209MiB/s (219MB/s-219MB/s), io=1465MiB (1536MB), run=7001-7001msec

$ rm /tmp/fio-output_512_8_io_uring_cmd-xnvme_2.txt
rm: cannot remove '/tmp/fio-output_512_8_io_uring_cmd-xnvme_2.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring_cmd-xnvme" --rw="randread" --size="12G" --iodepth="8" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring_cmd" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_512_8_io_uring_cmd-xnvme_2.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_8_io_uring_cmd-xnvme_2.txt
io_uring_cmd-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666790652,
  "timestamp_ms" : 1666790652800,
  "time" : "Wed Oct 26 15:24:12 2022",
  "jobs" : [
    {
      "jobname" : "io_uring_cmd-xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring_cmd-xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "8",
        "bs" : "512",
        "direct" : "0",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
        "xnvme_be" : "linux",
        "xnvme_async" : "io_uring_cmd",
        "xnvme_sync" : "nvme",
        "xnvme_admin" : "nvme",
        "xnvme_dev_nsid" : "1",
        "filename" : "/dev/ng0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 1528946176,
        "io_kbytes" : 1493111,
        "bw_bytes" : 218389683,
        "bw" : 213271,
        "iops" : 426541.208399,
        "runtime" : 7001,
        "total_ios" : 2986215,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1222,
          "max" : 37791,
          "mean" : 1814.158787,
          "stddev" : 388.864751,
          "N" : 2986216
        },
        "clat_ns" : {
          "min" : 121,
          "max" : 127819,
          "mean" : 16622.123058,
          "stddev" : 3687.178497,
          "N" : 2986222,
          "percentile" : {
            "1.000000" : 12608,
            "5.000000" : 13248,
            "10.000000" : 14144,
            "20.000000" : 15296,
            "30.000000" : 15680,
            "40.000000" : 16064,
            "50.000000" : 16192,
            "60.000000" : 16512,
            "70.000000" : 16768,
            "80.000000" : 17280,
            "90.000000" : 17536,
            "95.000000" : 18560,
            "99.000000" : 36608,
            "99.500000" : 38656,
            "99.900000" : 48896,
            "99.950000" : 65280,
            "99.990000" : 80384
          }
        },
        "lat_ns" : {
          "min" : 6693,
          "max" : 129142,
          "mean" : 18436.281854,
          "stddev" : 3660.248095,
          "N" : 2986222
        },
        "bw_min" : 212347,
        "bw_max" : 214606,
        "bw_agg" : 100.000000,
        "bw_mean" : 213309.142857,
        "bw_dev" : 669.714733,
        "bw_samples" : 14,
        "iops_min" : 424694,
        "iops_max" : 429212,
        "iops_mean" : 426618.285714,
        "iops_stddev" : 1339.429467,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 21.257143,
      "sys_cpu" : 75.814286,
      "ctx" : 14684,
      "majf" : 0,
      "minf" : 1,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 100.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 99.999983,
        "8" : 0.100000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.037707,
        "20" : 96.299061,
        "50" : 3.567292,
        "100" : 0.095405,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 8,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring_cmd-xnvme: (groupid=0, jobs=1): err= 0: pid=10313: Wed Oct 26 15:24:12 2022
  read: IOPS=427k, BW=208MiB/s (218MB/s)(1458MiB/7001msec)
    slat (nsec): min=1222, max=37791, avg=1814.16, stdev=388.86
    clat (nsec): min=121, max=127819, avg=16622.12, stdev=3687.18
     lat (usec): min=6, max=129, avg=18.44, stdev= 3.66
    clat percentiles (nsec):
     |  1.00th=[12608],  5.00th=[13248], 10.00th=[14144], 20.00th=[15296],
     | 30.00th=[15680], 40.00th=[16064], 50.00th=[16192], 60.00th=[16512],
     | 70.00th=[16768], 80.00th=[17280], 90.00th=[17536], 95.00th=[18560],
     | 99.00th=[36608], 99.50th=[38656], 99.90th=[48896], 99.95th=[65280],
     | 99.99th=[80384]
   bw (  KiB/s): min=212347, max=214606, per=100.00%, avg=213309.14, stdev=669.71, samples=14
   iops        : min=424694, max=429212, avg=426618.29, stdev=1339.43, samples=14
  lat (nsec)   : 250=0.01%
  lat (usec)   : 10=0.04%, 20=96.30%, 50=3.57%, 100=0.10%, 250=0.01%
  cpu          : usr=21.26%, sys=75.81%, ctx=14684, majf=0, minf=1
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=2986215,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
   READ: bw=208MiB/s (218MB/s), 208MiB/s-208MiB/s (218MB/s-218MB/s), io=1458MiB (1529MB), run=7001-7001msec

$ rm /tmp/fio-output_512_8_io_uring_cmd-bdev_xnvme_0.txt
rm: cannot remove '/tmp/fio-output_512_8_io_uring_cmd-bdev_xnvme_0.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring_cmd-bdev_xnvme" --rw="randread" --size="12G" --iodepth="8" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_8_io_uring_cmd-bdev_xnvme_0.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_512_8_io_uring_cmd-bdev_xnvme_0.txt
io_uring_cmd-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666790667,
  "timestamp_ms" : 1666790667990,
  "time" : "Wed Oct 26 15:24:27 2022",
  "jobs" : [
    {
      "jobname" : "io_uring_cmd-bdev_xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring_cmd-bdev_xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "8",
        "bs" : "512",
        "direct" : "0",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",
        "filename" : "Nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 2123516928,
        "io_kbytes" : 2073747,
        "bw_bytes" : 303316230,
        "bw" : 296207,
        "iops" : 592413.512355,
        "runtime" : 7001,
        "total_ios" : 4147487,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 190,
          "max" : 26219,
          "mean" : 278.529163,
          "stddev" : 75.839500,
          "N" : 4147488
        },
        "clat_ns" : {
          "min" : 691,
          "max" : 131106,
          "mean" : 12536.847786,
          "stddev" : 4087.236657,
          "N" : 4147494,
          "percentile" : {
            "1.000000" : 7008,
            "5.000000" : 7904,
            "10.000000" : 8640,
            "20.000000" : 9664,
            "30.000000" : 10688,
            "40.000000" : 11456,
            "50.000000" : 12096,
            "60.000000" : 12608,
            "70.000000" : 13504,
            "80.000000" : 14656,
            "90.000000" : 15936,
            "95.000000" : 17280,
            "99.000000" : 29568,
            "99.500000" : 32128,
            "99.900000" : 43776,
            "99.950000" : 62720,
            "99.990000" : 76288
          }
        },
        "lat_ns" : {
          "min" : 5720,
          "max" : 131396,
          "mean" : 12815.376954,
          "stddev" : 4084.439124,
          "N" : 4147494
        },
        "bw_min" : 295554,
        "bw_max" : 296921,
        "bw_agg" : 100.000000,
        "bw_mean" : 296335.857143,
        "bw_dev" : 340.016483,
        "bw_samples" : 14,
        "iops_min" : 591108,
        "iops_max" : 593843,
        "iops_mean" : 592671.857143,
        "iops_stddev" : 680.086969,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 99.985714,
      "sys_cpu" : 0.000000,
      "ctx" : 21,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.100000,
        "2" : 0.100000,
        "4" : 35.838135,
        "8" : 64.137537,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 99.802206,
        "8" : 0.197794,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.010000,
        "1000" : 0.010000
      },
      "latency_us" : {
        "2" : 0.010000,
        "4" : 0.010000,
        "10" : 23.339482,
        "20" : 73.205245,
        "50" : 3.369510,
        "100" : 0.084557,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 8,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring_cmd-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=10371: Wed Oct 26 15:24:27 2022
  read: IOPS=592k, BW=289MiB/s (303MB/s)(2025MiB/7001msec)
    slat (nsec): min=190, max=26219, avg=278.53, stdev=75.84
    clat (nsec): min=691, max=131106, avg=12536.85, stdev=4087.24
     lat (usec): min=5, max=131, avg=12.82, stdev= 4.08
    clat percentiles (nsec):
     |  1.00th=[ 7008],  5.00th=[ 7904], 10.00th=[ 8640], 20.00th=[ 9664],
     | 30.00th=[10688], 40.00th=[11456], 50.00th=[12096], 60.00th=[12608],
     | 70.00th=[13504], 80.00th=[14656], 90.00th=[15936], 95.00th=[17280],
     | 99.00th=[29568], 99.50th=[32128], 99.90th=[43776], 99.95th=[62720],
     | 99.99th=[76288]
   bw (  KiB/s): min=295554, max=296921, per=100.00%, avg=296335.86, stdev=340.02, samples=14
   iops        : min=591108, max=593843, avg=592671.86, stdev=680.09, samples=14
  lat (nsec)   : 750=0.01%, 1000=0.01%
  lat (usec)   : 2=0.01%, 4=0.01%, 10=23.34%, 20=73.21%, 50=3.37%
  lat (usec)   : 100=0.08%, 250=0.01%
  cpu          : usr=99.99%, sys=0.00%, ctx=21, majf=0, minf=0
  IO depths    : 1=0.1%, 2=0.1%, 4=35.8%, 8=64.1%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=99.8%, 8=0.2%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=4147487,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
   READ: bw=289MiB/s (303MB/s), 289MiB/s-289MiB/s (303MB/s-303MB/s), io=2025MiB (2124MB), run=7001-7001msec

$ rm /tmp/fio-output_512_8_io_uring_cmd-bdev_xnvme_1.txt
rm: cannot remove '/tmp/fio-output_512_8_io_uring_cmd-bdev_xnvme_1.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring-reference" --rw="randread" --size="12G" --iodepth="1" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_1_io_uring-reference_1.txt" --cpus_allowed="1"
$ /opt/fio/bin/fio --name="io_uring_cmd-bdev_xnvme" --rw="randread" --size="12G" --iodepth="8" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_8_io_uring_cmd-bdev_xnvme_1.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_512_8_io_uring_cmd-bdev_xnvme_1.txt
io_uring_cmd-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666790683,
  "timestamp_ms" : 1666790683397,
  "time" : "Wed Oct 26 15:24:43 2022",
  "jobs" : [
    {
      "jobname" : "io_uring_cmd-bdev_xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring_cmd-bdev_xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "8",
        "bs" : "512",
        "direct" : "0",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",
        "filename" : "Nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 1576720896,
        "io_kbytes" : 1539766,
        "bw_bytes" : 225213668,
        "bw" : 219935,
        "iops" : 439869.447222,
        "runtime" : 7001,
        "total_ios" : 3079526,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 170,
          "max" : 30026,
          "mean" : 708.241025,
          "stddev" : 279.522443,
          "N" : 3079527
        },
        "clat_ns" : {
          "min" : 711,
          "max" : 200556,
          "mean" : 16166.595282,
          "stddev" : 5857.765495,
          "N" : 3079533,
          "percentile" : {
            "1.000000" : 7456,
            "5.000000" : 8768,
            "10.000000" : 9408,
            "20.000000" : 11072,
            "30.000000" : 12480,
            "40.000000" : 13632,
            "50.000000" : 15168,
            "60.000000" : 16768,
            "70.000000" : 18816,
            "80.000000" : 21376,
            "90.000000" : 23936,
            "95.000000" : 25728,
            "99.000000" : 33024,
            "99.500000" : 36096,
            "99.900000" : 47360,
            "99.950000" : 64256,
            "99.990000" : 77312
          }
        },
        "lat_ns" : {
          "min" : 5370,
          "max" : 202119,
          "mean" : 16874.835481,
          "stddev" : 5927.145293,
          "N" : 3079533
        },
        "bw_min" : 198922,
        "bw_max" : 296648,
        "bw_agg" : 100.000000,
        "bw_mean" : 220001.714286,
        "bw_dev" : 36653.514516,
        "bw_samples" : 14,
        "iops_min" : 397844,
        "iops_max" : 593296,
        "iops_mean" : 440003.428571,
        "iops_stddev" : 73307.214011,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 99.985714,
      "sys_cpu" : 0.000000,
      "ctx" : 21,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.100000,
        "2" : 3.569933,
        "4" : 45.009070,
        "8" : 51.419309,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 97.162062,
        "8" : 2.837938,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.010000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.010000,
        "4" : 0.000000,
        "10" : 13.485322,
        "20" : 61.459426,
        "50" : 24.963355,
        "100" : 0.090858,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 8,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring_cmd-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=10431: Wed Oct 26 15:24:43 2022
  read: IOPS=440k, BW=215MiB/s (225MB/s)(1504MiB/7001msec)
    slat (nsec): min=170, max=30026, avg=708.24, stdev=279.52
    clat (nsec): min=711, max=200556, avg=16166.60, stdev=5857.77
     lat (usec): min=5, max=202, avg=16.87, stdev= 5.93
    clat percentiles (nsec):
     |  1.00th=[ 7456],  5.00th=[ 8768], 10.00th=[ 9408], 20.00th=[11072],
     | 30.00th=[12480], 40.00th=[13632], 50.00th=[15168], 60.00th=[16768],
     | 70.00th=[18816], 80.00th=[21376], 90.00th=[23936], 95.00th=[25728],
     | 99.00th=[33024], 99.50th=[36096], 99.90th=[47360], 99.95th=[64256],
     | 99.99th=[77312]
   bw (  KiB/s): min=198922, max=296648, per=100.00%, avg=220001.71, stdev=36653.51, samples=14
   iops        : min=397844, max=593296, avg=440003.43, stdev=73307.21, samples=14
  lat (nsec)   : 750=0.01%
  lat (usec)   : 2=0.01%, 10=13.49%, 20=61.46%, 50=24.96%, 100=0.09%
  lat (usec)   : 250=0.01%
  cpu          : usr=99.99%, sys=0.00%, ctx=21, majf=0, minf=0
  IO depths    : 1=0.1%, 2=3.6%, 4=45.0%, 8=51.4%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=97.2%, 8=2.8%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=3079526,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
   READ: bw=215MiB/s (225MB/s), 215MiB/s-215MiB/s (225MB/s-225MB/s), io=1504MiB (1577MB), run=7001-7001msec

$ rm /tmp/fio-output_512_8_io_uring_cmd-bdev_xnvme_2.txt
rm: cannot remove '/tmp/fio-output_512_8_io_uring_cmd-bdev_xnvme_2.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring_cmd-bdev_xnvme" --rw="randread" --size="12G" --iodepth="8" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_8_io_uring_cmd-bdev_xnvme_2.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_512_8_io_uring_cmd-bdev_xnvme_2.txt
io_uring_cmd-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666790698,
  "timestamp_ms" : 1666790698756,
  "time" : "Wed Oct 26 15:24:58 2022",
  "jobs" : [
    {
      "jobname" : "io_uring_cmd-bdev_xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring_cmd-bdev_xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "8",
        "bs" : "512",
        "direct" : "0",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",
        "filename" : "Nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 1556664832,
        "io_kbytes" : 1520180,
        "bw_bytes" : 222348926,
        "bw" : 217137,
        "iops" : 434274.246536,
        "runtime" : 7001,
        "total_ios" : 3040354,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 631,
          "max" : 14688,
          "mean" : 866.610232,
          "stddev" : 144.688161,
          "N" : 3040354
        },
        "clat_ns" : {
          "min" : 6462,
          "max" : 132108,
          "mean" : 16360.181632,
          "stddev" : 5137.436929,
          "N" : 3040360,
          "percentile" : {
            "1.000000" : 8384,
            "5.000000" : 9536,
            "10.000000" : 10304,
            "20.000000" : 12224,
            "30.000000" : 13376,
            "40.000000" : 14528,
            "50.000000" : 15680,
            "60.000000" : 17024,
            "70.000000" : 18816,
            "80.000000" : 20096,
            "90.000000" : 22144,
            "95.000000" : 24960,
            "99.000000" : 32640,
            "99.500000" : 36096,
            "99.900000" : 44800,
            "99.950000" : 61696,
            "99.990000" : 76288
          }
        },
        "lat_ns" : {
          "min" : 7364,
          "max" : 132779,
          "mean" : 17226.791887,
          "stddev" : 5119.625326,
          "N" : 3040360
        },
        "bw_min" : 216825,
        "bw_max" : 217381,
        "bw_agg" : 100.000000,
        "bw_mean" : 217176.142857,
        "bw_dev" : 164.800045,
        "bw_samples" : 14,
        "iops_min" : 433650,
        "iops_max" : 434762,
        "iops_mean" : 434352.357143,
        "iops_stddev" : 329.733134,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 99.985714,
      "sys_cpu" : 0.000000,
      "ctx" : 16,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.100000,
        "2" : 1.185553,
        "4" : 45.805258,
        "8" : 53.008959,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 98.136979,
        "8" : 1.863021,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 8.138000,
        "20" : 70.711042,
        "50" : 21.071198,
        "100" : 0.078215,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 8,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring_cmd-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=10490: Wed Oct 26 15:24:58 2022
  read: IOPS=434k, BW=212MiB/s (222MB/s)(1485MiB/7001msec)
    slat (nsec): min=631, max=14688, avg=866.61, stdev=144.69
    clat (usec): min=6, max=132, avg=16.36, stdev= 5.14
     lat (usec): min=7, max=132, avg=17.23, stdev= 5.12
    clat percentiles (nsec):
     |  1.00th=[ 8384],  5.00th=[ 9536], 10.00th=[10304], 20.00th=[12224],
     | 30.00th=[13376], 40.00th=[14528], 50.00th=[15680], 60.00th=[17024],
     | 70.00th=[18816], 80.00th=[20096], 90.00th=[22144], 95.00th=[24960],
     | 99.00th=[32640], 99.50th=[36096], 99.90th=[44800], 99.95th=[61696],
     | 99.99th=[76288]
   bw (  KiB/s): min=216825, max=217381, per=100.00%, avg=217176.14, stdev=164.80, samples=14
   iops        : min=433650, max=434762, avg=434352.36, stdev=329.73, samples=14
  lat (usec)   : 10=8.14%, 20=70.71%, 50=21.07%, 100=0.08%, 250=0.01%
  cpu          : usr=99.99%, sys=0.00%, ctx=16, majf=0, minf=0
  IO depths    : 1=0.1%, 2=1.2%, 4=45.8%, 8=53.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=98.1%, 8=1.9%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=3040354,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
   READ: bw=212MiB/s (222MB/s), 212MiB/s-212MiB/s (222MB/s-222MB/s), io=1485MiB (1557MB), run=7001-7001msec

$ rm /tmp/fio-output_512_8_io_uring-reference_0.txt
rm: cannot remove '/tmp/fio-output_512_8_io_uring-reference_0.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring-reference" --rw="randread" --size="12G" --iodepth="8" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_8_io_uring-reference_0.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_8_io_uring-reference_0.txt
io_uring-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=io_uring, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666790712,
  "timestamp_ms" : 1666790712555,
  "time" : "Wed Oct 26 15:25:12 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-reference",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-reference",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "8",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "io_uring",
        "filename" : "/dev/nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 1552151040,
        "io_kbytes" : 1515772,
        "bw_bytes" : 221704190,
        "bw" : 216507,
        "iops" : 433014.855021,
        "runtime" : 7001,
        "total_ios" : 3031537,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1332,
          "max" : 31810,
          "mean" : 1862.089115,
          "stddev" : 370.306038,
          "N" : 3031538
        },
        "clat_ns" : {
          "min" : 2144,
          "max" : 123952,
          "mean" : 16334.740680,
          "stddev" : 3609.143862,
          "N" : 3031544,
          "percentile" : {
            "1.000000" : 12480,
            "5.000000" : 12992,
            "10.000000" : 13248,
            "20.000000" : 14784,
            "30.000000" : 15296,
            "40.000000" : 15552,
            "50.000000" : 15936,
            "60.000000" : 16192,
            "70.000000" : 16768,
            "80.000000" : 17280,
            "90.000000" : 17792,
            "95.000000" : 18816,
            "99.000000" : 32640,
            "99.500000" : 35072,
            "99.900000" : 51456,
            "99.950000" : 67072,
            "99.990000" : 78336
          }
        },
        "lat_ns" : {
          "min" : 7133,
          "max" : 125345,
          "mean" : 18196.829907,
          "stddev" : 3573.540743,
          "N" : 3031544
        },
        "bw_min" : 215528,
        "bw_max" : 217965,
        "bw_agg" : 100.000000,
        "bw_mean" : 216654.571429,
        "bw_dev" : 805.018749,
        "bw_samples" : 14,
        "iops_min" : 431056,
        "iops_max" : 435931,
        "iops_mean" : 433309.285714,
        "iops_stddev" : 1610.180087,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 16.757143,
      "sys_cpu" : 80.328571,
      "ctx" : 15022,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 100.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 99.999967,
        "8" : 0.100000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010000,
        "10" : 0.056506,
        "20" : 95.943741,
        "50" : 3.889281,
        "100" : 0.109647,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 8,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ],
  "disk_util" : [
    {
      "name" : "nvme0n1",
      "read_ios" : 4373344,
      "write_ios" : 0,
      "read_merges" : 0,
      "write_merges" : 0,
      "read_ticks" : 25161,
      "write_ticks" : 0,
      "in_queue" : 25161,
      "util" : 99.108381
    }
  ]
}

io_uring-reference: (groupid=0, jobs=1): err= 0: pid=10537: Wed Oct 26 15:25:12 2022
  read: IOPS=433k, BW=211MiB/s (222MB/s)(1480MiB/7001msec)
    slat (nsec): min=1332, max=31810, avg=1862.09, stdev=370.31
    clat (usec): min=2, max=123, avg=16.33, stdev= 3.61
     lat (usec): min=7, max=125, avg=18.20, stdev= 3.57
    clat percentiles (nsec):
     |  1.00th=[12480],  5.00th=[12992], 10.00th=[13248], 20.00th=[14784],
     | 30.00th=[15296], 40.00th=[15552], 50.00th=[15936], 60.00th=[16192],
     | 70.00th=[16768], 80.00th=[17280], 90.00th=[17792], 95.00th=[18816],
     | 99.00th=[32640], 99.50th=[35072], 99.90th=[51456], 99.95th=[67072],
     | 99.99th=[78336]
   bw (  KiB/s): min=215528, max=217965, per=100.00%, avg=216654.57, stdev=805.02, samples=14
   iops        : min=431056, max=435931, avg=433309.29, stdev=1610.18, samples=14
  lat (usec)   : 4=0.01%, 10=0.06%, 20=95.94%, 50=3.89%, 100=0.11%
  lat (usec)   : 250=0.01%
  cpu          : usr=16.76%, sys=80.33%, ctx=15022, majf=0, minf=0
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=3031537,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
   READ: bw=211MiB/s (222MB/s), 211MiB/s-211MiB/s (222MB/s-222MB/s), io=1480MiB (1552MB), run=7001-7001msec

Disk stats (read/write):
  nvme0n1: ios=4373344/0, merge=0/0, ticks=25161/0, in_queue=25161, util=99.11%

$ rm /tmp/fio-output_512_8_io_uring-reference_1.txt
rm: cannot remove '/tmp/fio-output_512_8_io_uring-reference_1.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring-reference" --rw="randread" --size="12G" --iodepth="8" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_8_io_uring-reference_1.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_1_io_uring-reference_1.txt
io_uring-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=io_uring, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666789291,
  "timestamp_ms" : 1666789291136,
  "time" : "Wed Oct 26 15:01:31 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-reference",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-reference",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "1",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "io_uring",
        "filename" : "/dev/nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 392546816,
        "io_kbytes" : 383346,
        "bw_bytes" : 56070106,
        "bw" : 54755,
        "iops" : 109511.784031,
        "runtime" : 7001,
        "total_ios" : 766692,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1452,
          "max" : 7204,
          "mean" : 1551.774812,
          "stddev" : 73.369744,
          "N" : 766693
        },
        "clat_ns" : {
          "min" : 1042,
          "max" : 78077,
          "mean" : 7262.353467,
          "stddev" : 979.853703,
          "N" : 766692,
          "percentile" : {
            "1.000000" : 6816,
            "5.000000" : 6944,
            "10.000000" : 7008,
            "20.000000" : 7072,
            "30.000000" : 7072,
            "40.000000" : 7136,
            "50.000000" : 7136,
            "60.000000" : 7200,
            "70.000000" : 7264,
            "80.000000" : 7264,
            "90.000000" : 7328,
            "95.000000" : 7456,
            "99.000000" : 10432,
            "99.500000" : 13632,
            "99.900000" : 15680,
            "99.950000" : 15808,
            "99.990000" : 58112
          }
        },
        "lat_ns" : {
          "min" : 6743,
          "max" : 79590,
          "mean" : 8814.128316,
          "stddev" : 984.427864,
          "N" : 766692
        },
        "bw_min" : 54595,
        "bw_max" : 55137,
        "bw_agg" : 100.000000,
        "bw_mean" : 54770.357143,
        "bw_dev" : 141.769915,
        "bw_samples" : 14,
        "iops_min" : 109190,
        "iops_max" : 110274,
        "iops_mean" : 109540.714286,
        "iops_stddev" : 283.539830,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 8.571429,
      "sys_cpu" : 34.971429,
      "ctx" : 766689,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 100.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.010000,
        "4" : 0.010000,
        "10" : 98.723738,
        "20" : 1.253959,
        "50" : 0.010000,
        "100" : 0.014217,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 1,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ],
  "disk_util" : [
    {
      "name" : "nvme0n1",
      "read_ios" : 1098692,
      "write_ios" : 0,
      "read_merges" : 0,
      "write_merges" : 0,
      "read_ticks" : 5507,
      "write_ticks" : 0,
      "in_queue" : 5507,
      "util" : 99.069123
    }
  ]
}

io_uring-reference: (groupid=0, jobs=1): err= 0: pid=5437: Wed Oct 26 15:01:31 2022
  read: IOPS=110k, BW=53.5MiB/s (56.1MB/s)(374MiB/7001msec)
    slat (nsec): min=1452, max=7204, avg=1551.77, stdev=73.37
    clat (nsec): min=1042, max=78077, avg=7262.35, stdev=979.85
     lat (nsec): min=6743, max=79590, avg=8814.13, stdev=984.43
    clat percentiles (nsec):
     |  1.00th=[ 6816],  5.00th=[ 6944], 10.00th=[ 7008], 20.00th=[ 7072],
     | 30.00th=[ 7072], 40.00th=[ 7136], 50.00th=[ 7136], 60.00th=[ 7200],
     | 70.00th=[ 7264], 80.00th=[ 7264], 90.00th=[ 7328], 95.00th=[ 7456],
     | 99.00th=[10432], 99.50th=[13632], 99.90th=[15680], 99.95th=[15808],
     | 99.99th=[58112]
   bw (  KiB/s): min=54595, max=55137, per=100.00%, avg=54770.36, stdev=141.77, samples=14
   iops        : min=109190, max=110274, avg=109540.71, stdev=283.54, samples=14
  lat (usec)   : 2=0.01%, 4=0.01%, 10=98.72%, 20=1.25%, 50=0.01%
  lat (usec)   : 100=0.01%
  cpu          : usr=8.57%, sys=34.97%, ctx=766689, majf=0, minf=0
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=766692,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=53.5MiB/s (56.1MB/s), 53.5MiB/s-53.5MiB/s (56.1MB/s-56.1MB/s), io=374MiB (393MB), run=7001-7001msec

Disk stats (read/write):
  nvme0n1: ios=1098692/0, merge=0/0, ticks=5507/0, in_queue=5507, util=99.07%

$ cat /tmp/fio-output_512_8_io_uring-reference_1.txt
io_uring-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=io_uring, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666790726,
  "timestamp_ms" : 1666790726420,
  "time" : "Wed Oct 26 15:25:26 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-reference",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-reference",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "8",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "io_uring",
        "filename" : "/dev/nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 1511606784,
        "io_kbytes" : 1476178,
        "bw_bytes" : 215912981,
        "bw" : 210852,
        "iops" : 421704.042280,
        "runtime" : 7001,
        "total_ios" : 2952350,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1333,
          "max" : 30808,
          "mean" : 1878.538164,
          "stddev" : 370.577820,
          "N" : 2952350
        },
        "clat_ns" : {
          "min" : 3927,
          "max" : 150492,
          "mean" : 16780.232220,
          "stddev" : 3545.257165,
          "N" : 2952357,
          "percentile" : {
            "1.000000" : 12736,
            "5.000000" : 13248,
            "10.000000" : 13760,
            "20.000000" : 15168,
            "30.000000" : 15808,
            "40.000000" : 16192,
            "50.000000" : 16512,
            "60.000000" : 16768,
            "70.000000" : 17024,
            "80.000000" : 17536,
            "90.000000" : 18048,
            "95.000000" : 18816,
            "99.000000" : 33024,
            "99.500000" : 35072,
            "99.900000" : 50944,
            "99.950000" : 67072,
            "99.990000" : 78336
          }
        },
        "lat_ns" : {
          "min" : 7194,
          "max" : 151965,
          "mean" : 18658.769599,
          "stddev" : 3523.940859,
          "N" : 2952357
        },
        "bw_min" : 208714,
        "bw_max" : 212848,
        "bw_agg" : 100.000000,
        "bw_mean" : 210911.285714,
        "bw_dev" : 1029.007022,
        "bw_samples" : 14,
        "iops_min" : 417428,
        "iops_max" : 425697,
        "iops_mean" : 421822.500000,
        "iops_stddev" : 2058.121277,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 15.842857,
      "sys_cpu" : 81.442857,
      "ctx" : 15090,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 100.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 99.999966,
        "8" : 0.100000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010000,
        "10" : 0.087591,
        "20" : 95.854726,
        "50" : 3.952004,
        "100" : 0.104730,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 8,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ],
  "disk_util" : [
    {
      "name" : "nvme0n1",
      "read_ios" : 4279306,
      "write_ios" : 0,
      "read_merges" : 0,
      "write_merges" : 0,
      "read_ticks" : 24601,
      "write_ticks" : 0,
      "in_queue" : 24601,
      "util" : 99.069123
    }
  ]
}

io_uring-reference: (groupid=0, jobs=1): err= 0: pid=10583: Wed Oct 26 15:25:26 2022
  read: IOPS=422k, BW=206MiB/s (216MB/s)(1442MiB/7001msec)
    slat (nsec): min=1333, max=30808, avg=1878.54, stdev=370.58
    clat (usec): min=3, max=150, avg=16.78, stdev= 3.55
     lat (usec): min=7, max=151, avg=18.66, stdev= 3.52
    clat percentiles (nsec):
     |  1.00th=[12736],  5.00th=[13248], 10.00th=[13760], 20.00th=[15168],
     | 30.00th=[15808], 40.00th=[16192], 50.00th=[16512], 60.00th=[16768],
     | 70.00th=[17024], 80.00th=[17536], 90.00th=[18048], 95.00th=[18816],
     | 99.00th=[33024], 99.50th=[35072], 99.90th=[50944], 99.95th=[67072],
     | 99.99th=[78336]
   bw (  KiB/s): min=208714, max=212848, per=100.00%, avg=210911.29, stdev=1029.01, samples=14
   iops        : min=417428, max=425697, avg=421822.50, stdev=2058.12, samples=14
  lat (usec)   : 4=0.01%, 10=0.09%, 20=95.85%, 50=3.95%, 100=0.10%
  lat (usec)   : 250=0.01%
  cpu          : usr=15.84%, sys=81.44%, ctx=15090, majf=0, minf=0
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=2952350,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
   READ: bw=206MiB/s (216MB/s), 206MiB/s-206MiB/s (216MB/s-216MB/s), io=1442MiB (1512MB), run=7001-7001msec

Disk stats (read/write):
  nvme0n1: ios=4279306/0, merge=0/0, ticks=24601/0, in_queue=24601, util=99.07%

$ rm /tmp/fio-output_512_8_io_uring-reference_2.txt
rm: cannot remove '/tmp/fio-output_512_8_io_uring-reference_2.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring-reference" --rw="randread" --size="12G" --iodepth="8" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_8_io_uring-reference_2.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_8_io_uring-reference_2.txt
io_uring-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=io_uring, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666790740,
  "timestamp_ms" : 1666790740160,
  "time" : "Wed Oct 26 15:25:40 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-reference",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-reference",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "8",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "io_uring",
        "filename" : "/dev/nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 1562410496,
        "io_kbytes" : 1525791,
        "bw_bytes" : 223169618,
        "bw" : 217939,
        "iops" : 435877.017569,
        "runtime" : 7001,
        "total_ios" : 3051575,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1322,
          "max" : 37350,
          "mean" : 1848.436398,
          "stddev" : 360.896377,
          "N" : 3051576
        },
        "clat_ns" : {
          "min" : 5079,
          "max" : 131407,
          "mean" : 16231.104474,
          "stddev" : 3574.667520,
          "N" : 3051582,
          "percentile" : {
            "1.000000" : 12480,
            "5.000000" : 12992,
            "10.000000" : 13248,
            "20.000000" : 14784,
            "30.000000" : 15296,
            "40.000000" : 15424,
            "50.000000" : 15808,
            "60.000000" : 16064,
            "70.000000" : 16512,
            "80.000000" : 17280,
            "90.000000" : 17792,
            "95.000000" : 18560,
            "99.000000" : 32640,
            "99.500000" : 34560,
            "99.900000" : 49408,
            "99.950000" : 65280,
            "99.990000" : 77312
          }
        },
        "lat_ns" : {
          "min" : 6452,
          "max" : 132769,
          "mean" : 18079.540984,
          "stddev" : 3537.875972,
          "N" : 3051582
        },
        "bw_min" : 217144,
        "bw_max" : 219141,
        "bw_agg" : 100.000000,
        "bw_mean" : 217999.642857,
        "bw_dev" : 593.977934,
        "bw_samples" : 14,
        "iops_min" : 434288,
        "iops_max" : 438282,
        "iops_mean" : 435999.357143,
        "iops_stddev" : 1188.013701,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 16.628571,
      "sys_cpu" : 80.442857,
      "ctx" : 14900,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 100.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 99.999967,
        "8" : 0.100000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.058167,
        "20" : 96.003277,
        "50" : 3.840115,
        "100" : 0.098146,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 8,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ],
  "disk_util" : [
    {
      "name" : "nvme0n1",
      "read_ios" : 4399333,
      "write_ios" : 0,
      "read_merges" : 0,
      "write_merges" : 0,
      "read_ticks" : 25278,
      "write_ticks" : 0,
      "in_queue" : 25278,
      "util" : 99.069123
    }
  ]
}

io_uring-reference: (groupid=0, jobs=1): err= 0: pid=10630: Wed Oct 26 15:25:40 2022
  read: IOPS=436k, BW=213MiB/s (223MB/s)(1490MiB/7001msec)
    slat (nsec): min=1322, max=37350, avg=1848.44, stdev=360.90
    clat (usec): min=5, max=131, avg=16.23, stdev= 3.57
     lat (usec): min=6, max=132, avg=18.08, stdev= 3.54
    clat percentiles (nsec):
     |  1.00th=[12480],  5.00th=[12992], 10.00th=[13248], 20.00th=[14784],
     | 30.00th=[15296], 40.00th=[15424], 50.00th=[15808], 60.00th=[16064],
     | 70.00th=[16512], 80.00th=[17280], 90.00th=[17792], 95.00th=[18560],
     | 99.00th=[32640], 99.50th=[34560], 99.90th=[49408], 99.95th=[65280],
     | 99.99th=[77312]
   bw (  KiB/s): min=217144, max=219141, per=100.00%, avg=217999.64, stdev=593.98, samples=14
   iops        : min=434288, max=438282, avg=435999.36, stdev=1188.01, samples=14
  lat (usec)   : 10=0.06%, 20=96.00%, 50=3.84%, 100=0.10%, 250=0.01%
  cpu          : usr=16.63%, sys=80.44%, ctx=14900, majf=0, minf=0
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=3051575,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
   READ: bw=213MiB/s (223MB/s), 213MiB/s-213MiB/s (223MB/s-223MB/s), io=1490MiB (1562MB), run=7001-7001msec

Disk stats (read/write):
  nvme0n1: ios=4399333/0, merge=0/0, ticks=25278/0, in_queue=25278, util=99.07%

$ rm /tmp/fio-output_512_8_io_uring-bdev_uring_0.txt
rm: cannot remove '/tmp/fio-output_512_8_io_uring-bdev_uring_0.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring-bdev_uring" --rw="randread" --size="12G" --iodepth="8" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_uring_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_8_io_uring-bdev_uring_0.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_512_8_io_uring-bdev_uring_0.txt
io_uring-bdev_uring: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666790755,
  "timestamp_ms" : 1666790755304,
  "time" : "Wed Oct 26 15:25:55 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-bdev_uring",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-bdev_uring",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "8",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_uring_foo.conf",
        "filename" : "Nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 1789819392,
        "io_kbytes" : 1747870,
        "bw_bytes" : 255651962,
        "bw" : 249660,
        "iops" : 499319.240109,
        "runtime" : 7001,
        "total_ios" : 3495734,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 110,
          "max" : 23364,
          "mean" : 152.689076,
          "stddev" : 114.264283,
          "N" : 3495734
        },
        "clat_ns" : {
          "min" : 6803,
          "max" : 131567,
          "mean" : 15002.125793,
          "stddev" : 4088.111547,
          "N" : 3495738,
          "percentile" : {
            "1.000000" : 9152,
            "5.000000" : 12608,
            "10.000000" : 12864,
            "20.000000" : 13248,
            "30.000000" : 13504,
            "40.000000" : 13760,
            "50.000000" : 14016,
            "60.000000" : 14272,
            "70.000000" : 14656,
            "80.000000" : 15296,
            "90.000000" : 18304,
            "95.000000" : 21632,
            "99.000000" : 33024,
            "99.500000" : 34560,
            "99.900000" : 51968,
            "99.950000" : 67072,
            "99.990000" : 78336
          }
        },
        "lat_ns" : {
          "min" : 6973,
          "max" : 131697,
          "mean" : 15154.814849,
          "stddev" : 4085.047977,
          "N" : 3495738
        },
        "bw_min" : 247874,
        "bw_max" : 252834,
        "bw_agg" : 100.000000,
        "bw_mean" : 249794.071429,
        "bw_dev" : 1473.101410,
        "bw_samples" : 14,
        "iops_min" : 495748,
        "iops_max" : 505668,
        "iops_mean" : 499588.285714,
        "iops_stddev" : 2946.178762,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 30.042857,
      "sys_cpu" : 69.928571,
      "ctx" : 8,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.100000,
        "2" : 0.100000,
        "4" : 72.377446,
        "8" : 27.523175,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 98.713086,
        "8" : 1.286914,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 1.834121,
        "20" : 91.377834,
        "50" : 6.677682,
        "100" : 0.109390,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 8,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring-bdev_uring: (groupid=0, jobs=1): err= 0: pid=10686: Wed Oct 26 15:25:55 2022
  read: IOPS=499k, BW=244MiB/s (256MB/s)(1707MiB/7001msec)
    slat (nsec): min=110, max=23364, avg=152.69, stdev=114.26
    clat (usec): min=6, max=131, avg=15.00, stdev= 4.09
     lat (usec): min=6, max=131, avg=15.15, stdev= 4.09
    clat percentiles (nsec):
     |  1.00th=[ 9152],  5.00th=[12608], 10.00th=[12864], 20.00th=[13248],
     | 30.00th=[13504], 40.00th=[13760], 50.00th=[14016], 60.00th=[14272],
     | 70.00th=[14656], 80.00th=[15296], 90.00th=[18304], 95.00th=[21632],
     | 99.00th=[33024], 99.50th=[34560], 99.90th=[51968], 99.95th=[67072],
     | 99.99th=[78336]
   bw (  KiB/s): min=247874, max=252834, per=100.00%, avg=249794.07, stdev=1473.10, samples=14
   iops        : min=495748, max=505668, avg=499588.29, stdev=2946.18, samples=14
  lat (usec)   : 10=1.83%, 20=91.38%, 50=6.68%, 100=0.11%, 250=0.01%
  cpu          : usr=30.04%, sys=69.93%, ctx=8, majf=0, minf=0
  IO depths    : 1=0.1%, 2=0.1%, 4=72.4%, 8=27.5%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=98.7%, 8=1.3%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=3495734,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
   READ: bw=244MiB/s (256MB/s), 244MiB/s-244MiB/s (256MB/s-256MB/s), io=1707MiB (1790MB), run=7001-7001msec

$ rm /tmp/fio-output_512_8_io_uring-bdev_uring_1.txt
rm: cannot remove '/tmp/fio-output_512_8_io_uring-bdev_uring_1.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring-bdev_uring" --rw="randread" --size="12G" --iodepth="8" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_uring_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_8_io_uring-bdev_uring_1.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_512_8_io_uring-bdev_uring_1.txt
io_uring-bdev_uring: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666790770,
  "timestamp_ms" : 1666790770427,
  "time" : "Wed Oct 26 15:26:10 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-bdev_uring",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-bdev_uring",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "8",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_uring_foo.conf",
        "filename" : "Nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 1791554560,
        "io_kbytes" : 1749565,
        "bw_bytes" : 255899808,
        "bw" : 249902,
        "iops" : 499803.313812,
        "runtime" : 7001,
        "total_ios" : 3499123,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 110,
          "max" : 24446,
          "mean" : 156.179399,
          "stddev" : 115.772836,
          "N" : 3499123
        },
        "clat_ns" : {
          "min" : 6753,
          "max" : 143268,
          "mean" : 14972.242013,
          "stddev" : 4066.042433,
          "N" : 3499127,
          "percentile" : {
            "1.000000" : 9152,
            "5.000000" : 12480,
            "10.000000" : 12864,
            "20.000000" : 13248,
            "30.000000" : 13504,
            "40.000000" : 13760,
            "50.000000" : 14016,
            "60.000000" : 14272,
            "70.000000" : 14656,
            "80.000000" : 15168,
            "90.000000" : 18304,
            "95.000000" : 21632,
            "99.000000" : 33024,
            "99.500000" : 34560,
            "99.900000" : 50944,
            "99.950000" : 67072,
            "99.990000" : 78336
          }
        },
        "lat_ns" : {
          "min" : 6923,
          "max" : 143469,
          "mean" : 15128.421388,
          "stddev" : 4063.050872,
          "N" : 3499127
        },
        "bw_min" : 247329,
        "bw_max" : 253134,
        "bw_agg" : 100.000000,
        "bw_mean" : 250039.500000,
        "bw_dev" : 1803.163577,
        "bw_samples" : 14,
        "iops_min" : 494658,
        "iops_max" : 506268,
        "iops_mean" : 500079.142857,
        "iops_stddev" : 3606.253582,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 29.642857,
      "sys_cpu" : 70.328571,
      "ctx" : 9,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.100000,
        "2" : 0.117515,
        "4" : 72.370934,
        "8" : 27.508121,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 98.752327,
        "8" : 1.247673,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 1.845948,
        "20" : 91.447028,
        "50" : 6.605484,
        "100" : 0.100740,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 8,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring-bdev_uring: (groupid=0, jobs=1): err= 0: pid=10743: Wed Oct 26 15:26:10 2022
  read: IOPS=500k, BW=244MiB/s (256MB/s)(1709MiB/7001msec)
    slat (nsec): min=110, max=24446, avg=156.18, stdev=115.77
    clat (usec): min=6, max=143, avg=14.97, stdev= 4.07
     lat (usec): min=6, max=143, avg=15.13, stdev= 4.06
    clat percentiles (nsec):
     |  1.00th=[ 9152],  5.00th=[12480], 10.00th=[12864], 20.00th=[13248],
     | 30.00th=[13504], 40.00th=[13760], 50.00th=[14016], 60.00th=[14272],
     | 70.00th=[14656], 80.00th=[15168], 90.00th=[18304], 95.00th=[21632],
     | 99.00th=[33024], 99.50th=[34560], 99.90th=[50944], 99.95th=[67072],
     | 99.99th=[78336]
   bw (  KiB/s): min=247329, max=253134, per=100.00%, avg=250039.50, stdev=1803.16, samples=14
   iops        : min=494658, max=506268, avg=500079.14, stdev=3606.25, samples=14
  lat (usec)   : 10=1.85%, 20=91.45%, 50=6.61%, 100=0.10%, 250=0.01%
  cpu          : usr=29.64%, sys=70.33%, ctx=9, majf=0, minf=0
  IO depths    : 1=0.1%, 2=0.1%, 4=72.4%, 8=27.5%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=98.8%, 8=1.2%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=3499123,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
   READ: bw=244MiB/s (256MB/s), 244MiB/s-244MiB/s (256MB/s-256MB/s), io=1709MiB (1792MB), run=7001-7001msec

$ rm /tmp/fio-output_512_1_io_uring-reference_2.txt
rm: cannot remove '/tmp/fio-output_512_1_io_uring-reference_2.txt': No such file or directory

$ rm /tmp/fio-output_512_8_io_uring-bdev_uring_2.txt
rm: cannot remove '/tmp/fio-output_512_8_io_uring-bdev_uring_2.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring-bdev_uring" --rw="randread" --size="12G" --iodepth="8" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_uring_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_8_io_uring-bdev_uring_2.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_512_8_io_uring-bdev_uring_2.txt
io_uring-bdev_uring: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666790788,
  "timestamp_ms" : 1666790788540,
  "time" : "Wed Oct 26 15:26:28 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-bdev_uring",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-bdev_uring",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "8",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_uring_foo.conf",
        "filename" : "Nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 1763398144,
        "io_kbytes" : 1722068,
        "bw_bytes" : 251878037,
        "bw" : 245974,
        "iops" : 491948.435938,
        "runtime" : 7001,
        "total_ios" : 3444131,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 100,
          "max" : 20659,
          "mean" : 159.195923,
          "stddev" : 134.215657,
          "N" : 3444132
        },
        "clat_ns" : {
          "min" : 6412,
          "max" : 125806,
          "mean" : 15197.289828,
          "stddev" : 4131.746276,
          "N" : 3444137,
          "percentile" : {
            "1.000000" : 8896,
            "5.000000" : 12608,
            "10.000000" : 12992,
            "20.000000" : 13376,
            "30.000000" : 13632,
            "40.000000" : 13888,
            "50.000000" : 14144,
            "60.000000" : 14528,
            "70.000000" : 14912,
            "80.000000" : 15552,
            "90.000000" : 18816,
            "95.000000" : 22400,
            "99.000000" : 33024,
            "99.500000" : 34560,
            "99.900000" : 51456,
            "99.950000" : 69120,
            "99.990000" : 80384
          }
        },
        "lat_ns" : {
          "min" : 6612,
          "max" : 126016,
          "mean" : 15356.485787,
          "stddev" : 4127.680316,
          "N" : 3444137
        },
        "bw_min" : 244609,
        "bw_max" : 248021,
        "bw_agg" : 100.000000,
        "bw_mean" : 246103.357143,
        "bw_dev" : 1102.233300,
        "bw_samples" : 14,
        "iops_min" : 489218,
        "iops_max" : 496042,
        "iops_mean" : 492206.785714,
        "iops_stddev" : 2204.746425,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 32.785714,
      "sys_cpu" : 67.185714,
      "ctx" : 12,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.100000,
        "2" : 0.129118,
        "4" : 72.261014,
        "8" : 27.606935,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 98.422303,
        "8" : 1.577697,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 1.778126,
        "20" : 90.700905,
        "50" : 7.415281,
        "100" : 0.104874,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 8,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring-bdev_uring: (groupid=0, jobs=1): err= 0: pid=10800: Wed Oct 26 15:26:28 2022
  read: IOPS=492k, BW=240MiB/s (252MB/s)(1682MiB/7001msec)
    slat (nsec): min=100, max=20659, avg=159.20, stdev=134.22
    clat (usec): min=6, max=125, avg=15.20, stdev= 4.13
     lat (usec): min=6, max=126, avg=15.36, stdev= 4.13
    clat percentiles (nsec):
     |  1.00th=[ 8896],  5.00th=[12608], 10.00th=[12992], 20.00th=[13376],
     | 30.00th=[13632], 40.00th=[13888], 50.00th=[14144], 60.00th=[14528],
     | 70.00th=[14912], 80.00th=[15552], 90.00th=[18816], 95.00th=[22400],
     | 99.00th=[33024], 99.50th=[34560], 99.90th=[51456], 99.95th=[69120],
     | 99.99th=[80384]
   bw (  KiB/s): min=244609, max=248021, per=100.00%, avg=246103.36, stdev=1102.23, samples=14
   iops        : min=489218, max=496042, avg=492206.79, stdev=2204.75, samples=14
  lat (usec)   : 10=1.78%, 20=90.70%, 50=7.42%, 100=0.10%, 250=0.01%
  cpu          : usr=32.79%, sys=67.19%, ctx=12, majf=0, minf=0
  IO depths    : 1=0.1%, 2=0.1%, 4=72.3%, 8=27.6%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=98.4%, 8=1.6%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=3444131,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
   READ: bw=240MiB/s (252MB/s), 240MiB/s-240MiB/s (252MB/s-252MB/s), io=1682MiB (1763MB), run=7001-7001msec

$ rm /tmp/fio-output_512_8_io_uring-bdev_xnvme_0.txt
rm: cannot remove '/tmp/fio-output_512_8_io_uring-bdev_xnvme_0.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring-bdev_xnvme" --rw="randread" --size="12G" --iodepth="8" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_8_io_uring-bdev_xnvme_0.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_512_8_io_uring-bdev_xnvme_0.txt
io_uring-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666790805,
  "timestamp_ms" : 1666790805897,
  "time" : "Wed Oct 26 15:26:45 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-bdev_xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-bdev_xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "8",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring.conf",
        "filename" : "Nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 1621752320,
        "io_kbytes" : 1583742,
        "bw_bytes" : 231645810,
        "bw" : 226216,
        "iops" : 452432.081131,
        "runtime" : 7001,
        "total_ios" : 3167477,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1563,
          "max" : 52558,
          "mean" : 1732.091699,
          "stddev" : 132.863734,
          "N" : 3167478
        },
        "clat_ns" : {
          "min" : 250,
          "max" : 126818,
          "mean" : 15699.602769,
          "stddev" : 2539.253016,
          "N" : 3167484,
          "percentile" : {
            "1.000000" : 14912,
            "5.000000" : 14912,
            "10.000000" : 15040,
            "20.000000" : 15040,
            "30.000000" : 15168,
            "40.000000" : 15168,
            "50.000000" : 15296,
            "60.000000" : 15296,
            "70.000000" : 15296,
            "80.000000" : 15424,
            "90.000000" : 15808,
            "95.000000" : 16192,
            "99.000000" : 28288,
            "99.500000" : 30592,
            "99.900000" : 43776,
            "99.950000" : 60672,
            "99.990000" : 74240
          }
        },
        "lat_ns" : {
          "min" : 6412,
          "max" : 128531,
          "mean" : 17431.694401,
          "stddev" : 2546.918829,
          "N" : 3167484
        },
        "bw_min" : 224958,
        "bw_max" : 227326,
        "bw_agg" : 100.000000,
        "bw_mean" : 226257.285714,
        "bw_dev" : 705.539339,
        "bw_samples" : 14,
        "iops_min" : 449916,
        "iops_max" : 454652,
        "iops_mean" : 452514.642857,
        "iops_stddev" : 1411.151936,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 30.385714,
      "sys_cpu" : 69.600000,
      "ctx" : 15,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 100.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 99.999984,
        "8" : 0.100000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.010000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.027404,
        "20" : 96.891659,
        "50" : 3.001000,
        "100" : 0.078675,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 8,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=10858: Wed Oct 26 15:26:45 2022
  read: IOPS=452k, BW=221MiB/s (232MB/s)(1547MiB/7001msec)
    slat (nsec): min=1563, max=52558, avg=1732.09, stdev=132.86
    clat (nsec): min=250, max=126818, avg=15699.60, stdev=2539.25
     lat (usec): min=6, max=128, avg=17.43, stdev= 2.55
    clat percentiles (nsec):
     |  1.00th=[14912],  5.00th=[14912], 10.00th=[15040], 20.00th=[15040],
     | 30.00th=[15168], 40.00th=[15168], 50.00th=[15296], 60.00th=[15296],
     | 70.00th=[15296], 80.00th=[15424], 90.00th=[15808], 95.00th=[16192],
     | 99.00th=[28288], 99.50th=[30592], 99.90th=[43776], 99.95th=[60672],
     | 99.99th=[74240]
   bw (  KiB/s): min=224958, max=227326, per=100.00%, avg=226257.29, stdev=705.54, samples=14
   iops        : min=449916, max=454652, avg=452514.64, stdev=1411.15, samples=14
  lat (nsec)   : 500=0.01%
  lat (usec)   : 10=0.03%, 20=96.89%, 50=3.00%, 100=0.08%, 250=0.01%
  cpu          : usr=30.39%, sys=69.60%, ctx=15, majf=0, minf=0
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=3167477,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
   READ: bw=221MiB/s (232MB/s), 221MiB/s-221MiB/s (232MB/s-232MB/s), io=1547MiB (1622MB), run=7001-7001msec

$ rm /tmp/fio-output_512_8_io_uring-bdev_xnvme_1.txt
rm: cannot remove '/tmp/fio-output_512_8_io_uring-bdev_xnvme_1.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring-bdev_xnvme" --rw="randread" --size="12G" --iodepth="8" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_8_io_uring-bdev_xnvme_1.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_512_8_io_uring-bdev_xnvme_1.txt
io_uring-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666790824,
  "timestamp_ms" : 1666790824648,
  "time" : "Wed Oct 26 15:27:04 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-bdev_xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-bdev_xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "8",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring.conf",
        "filename" : "Nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 1612016128,
        "io_kbytes" : 1574234,
        "bw_bytes" : 230255124,
        "bw" : 224858,
        "iops" : 449715.897729,
        "runtime" : 7001,
        "total_ios" : 3148461,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1573,
          "max" : 40045,
          "mean" : 1761.557793,
          "stddev" : 123.973748,
          "N" : 3148462
        },
        "clat_ns" : {
          "min" : 250,
          "max" : 120826,
          "mean" : 15794.247972,
          "stddev" : 2581.682894,
          "N" : 3148468,
          "percentile" : {
            "1.000000" : 14912,
            "5.000000" : 15040,
            "10.000000" : 15040,
            "20.000000" : 15168,
            "30.000000" : 15168,
            "40.000000" : 15296,
            "50.000000" : 15296,
            "60.000000" : 15424,
            "70.000000" : 15424,
            "80.000000" : 15808,
            "90.000000" : 16064,
            "95.000000" : 16192,
            "99.000000" : 28032,
            "99.500000" : 30592,
            "99.900000" : 49408,
            "99.950000" : 64768,
            "99.990000" : 75264
          }
        },
        "lat_ns" : {
          "min" : 5831,
          "max" : 122500,
          "mean" : 17555.805718,
          "stddev" : 2589.532167,
          "N" : 3148468
        },
        "bw_min" : 224054,
        "bw_max" : 227215,
        "bw_agg" : 100.000000,
        "bw_mean" : 224976.142857,
        "bw_dev" : 840.788902,
        "bw_samples" : 14,
        "iops_min" : 448108,
        "iops_max" : 454430,
        "iops_mean" : 449952.428571,
        "iops_stddev" : 1681.479647,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 31.585714,
      "sys_cpu" : 68.400000,
      "ctx" : 11,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 100.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 99.999984,
        "8" : 0.100000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.010000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.032619,
        "20" : 96.921004,
        "50" : 2.948266,
        "100" : 0.097254,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 8,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=10915: Wed Oct 26 15:27:04 2022
  read: IOPS=450k, BW=220MiB/s (230MB/s)(1537MiB/7001msec)
    slat (nsec): min=1573, max=40045, avg=1761.56, stdev=123.97
    clat (nsec): min=250, max=120826, avg=15794.25, stdev=2581.68
     lat (usec): min=5, max=122, avg=17.56, stdev= 2.59
    clat percentiles (nsec):
     |  1.00th=[14912],  5.00th=[15040], 10.00th=[15040], 20.00th=[15168],
     | 30.00th=[15168], 40.00th=[15296], 50.00th=[15296], 60.00th=[15424],
     | 70.00th=[15424], 80.00th=[15808], 90.00th=[16064], 95.00th=[16192],
     | 99.00th=[28032], 99.50th=[30592], 99.90th=[49408], 99.95th=[64768],
     | 99.99th=[75264]
   bw (  KiB/s): min=224054, max=227215, per=100.00%, avg=224976.14, stdev=840.79, samples=14
   iops        : min=448108, max=454430, avg=449952.43, stdev=1681.48, samples=14
  lat (nsec)   : 500=0.01%
  lat (usec)   : 10=0.03%, 20=96.92%, 50=2.95%, 100=0.10%, 250=0.01%
  cpu          : usr=31.59%, sys=68.40%, ctx=11, majf=0, minf=0
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=3148461,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
   READ: bw=220MiB/s (230MB/s), 220MiB/s-220MiB/s (230MB/s-230MB/s), io=1537MiB (1612MB), run=7001-7001msec

$ rm /tmp/fio-output_512_8_io_uring-bdev_xnvme_2.txt
rm: cannot remove '/tmp/fio-output_512_8_io_uring-bdev_xnvme_2.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring-reference" --rw="randread" --size="12G" --iodepth="1" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_1_io_uring-reference_2.txt" --cpus_allowed="1"
$ /opt/fio/bin/fio --name="io_uring-bdev_xnvme" --rw="randread" --size="12G" --iodepth="8" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_8_io_uring-bdev_xnvme_2.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_512_8_io_uring-bdev_xnvme_2.txt
io_uring-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666790841,
  "timestamp_ms" : 1666790841375,
  "time" : "Wed Oct 26 15:27:21 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-bdev_xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-bdev_xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "8",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring.conf",
        "filename" : "Nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 1616416768,
        "io_kbytes" : 1578532,
        "bw_bytes" : 230883697,
        "bw" : 225472,
        "iops" : 450943.579489,
        "runtime" : 7001,
        "total_ios" : 3157056,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1572,
          "max" : 17793,
          "mean" : 1750.871442,
          "stddev" : 124.108060,
          "N" : 3157057
        },
        "clat_ns" : {
          "min" : 260,
          "max" : 128782,
          "mean" : 15750.550326,
          "stddev" : 2540.179660,
          "N" : 3157063,
          "percentile" : {
            "1.000000" : 14912,
            "5.000000" : 15040,
            "10.000000" : 15040,
            "20.000000" : 15168,
            "30.000000" : 15168,
            "40.000000" : 15168,
            "50.000000" : 15296,
            "60.000000" : 15296,
            "70.000000" : 15424,
            "80.000000" : 15808,
            "90.000000" : 15936,
            "95.000000" : 16192,
            "99.000000" : 28288,
            "99.500000" : 30592,
            "99.900000" : 44288,
            "99.950000" : 63232,
            "99.990000" : 74240
          }
        },
        "lat_ns" : {
          "min" : 6081,
          "max" : 130635,
          "mean" : 17501.421643,
          "stddev" : 2547.987942,
          "N" : 3157063
        },
        "bw_min" : 224516,
        "bw_max" : 226843,
        "bw_agg" : 100.000000,
        "bw_mean" : 225679.230769,
        "bw_dev" : 917.421582,
        "bw_samples" : 13,
        "iops_min" : 449032,
        "iops_max" : 453686,
        "iops_mean" : 451358.615385,
        "iops_stddev" : 1835.033630,
        "iops_samples" : 13
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 31.128571,
      "sys_cpu" : 68.857143,
      "ctx" : 8,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 100.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 99.999984,
        "8" : 0.100000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.010000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010000,
        "10" : 0.031517,
        "20" : 96.933409,
        "50" : 2.949868,
        "100" : 0.084382,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 8,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=10974: Wed Oct 26 15:27:21 2022
  read: IOPS=451k, BW=220MiB/s (231MB/s)(1542MiB/7001msec)
    slat (nsec): min=1572, max=17793, avg=1750.87, stdev=124.11
    clat (nsec): min=260, max=128782, avg=15750.55, stdev=2540.18
     lat (usec): min=6, max=130, avg=17.50, stdev= 2.55
    clat percentiles (nsec):
     |  1.00th=[14912],  5.00th=[15040], 10.00th=[15040], 20.00th=[15168],
     | 30.00th=[15168], 40.00th=[15168], 50.00th=[15296], 60.00th=[15296],
     | 70.00th=[15424], 80.00th=[15808], 90.00th=[15936], 95.00th=[16192],
     | 99.00th=[28288], 99.50th=[30592], 99.90th=[44288], 99.95th=[63232],
     | 99.99th=[74240]
   bw (  KiB/s): min=224516, max=226843, per=100.00%, avg=225679.23, stdev=917.42, samples=13
   iops        : min=449032, max=453686, avg=451358.62, stdev=1835.03, samples=13
  lat (nsec)   : 500=0.01%
  lat (usec)   : 4=0.01%, 10=0.03%, 20=96.93%, 50=2.95%, 100=0.08%
  lat (usec)   : 250=0.01%
  cpu          : usr=31.13%, sys=68.86%, ctx=8, majf=0, minf=0
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=3157056,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
   READ: bw=220MiB/s (231MB/s), 220MiB/s-220MiB/s (231MB/s-231MB/s), io=1542MiB (1616MB), run=7001-7001msec

$ rm /tmp/fio-output_512_8_io_uring-xnvme_0.txt
rm: cannot remove '/tmp/fio-output_512_8_io_uring-xnvme_0.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring-xnvme" --rw="randread" --size="12G" --iodepth="8" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_8_io_uring-xnvme_0.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_8_io_uring-xnvme_0.txt
io_uring-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666790856,
  "timestamp_ms" : 1666790856790,
  "time" : "Wed Oct 26 15:27:36 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "8",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
        "xnvme_be" : "linux",
        "xnvme_async" : "io_uring",
        "xnvme_sync" : "nvme",
        "xnvme_admin" : "nvme",
        "xnvme_dev_nsid" : "1",
        "filename" : "/dev/nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 1419834368,
        "io_kbytes" : 1386557,
        "bw_bytes" : 202804509,
        "bw" : 198051,
        "iops" : 396101.414084,
        "runtime" : 7001,
        "total_ios" : 2773106,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1393,
          "max" : 70382,
          "mean" : 2032.974013,
          "stddev" : 315.234496,
          "N" : 2773107
        },
        "clat_ns" : {
          "min" : 4167,
          "max" : 134191,
          "mean" : 17867.042998,
          "stddev" : 3010.820945,
          "N" : 2773113,
          "percentile" : {
            "1.000000" : 13376,
            "5.000000" : 15424,
            "10.000000" : 16768,
            "20.000000" : 17024,
            "30.000000" : 17280,
            "40.000000" : 17536,
            "50.000000" : 17536,
            "60.000000" : 17536,
            "70.000000" : 18048,
            "80.000000" : 18048,
            "90.000000" : 18304,
            "95.000000" : 19584,
            "99.000000" : 31616,
            "99.500000" : 34560,
            "99.900000" : 50432,
            "99.950000" : 64256,
            "99.990000" : 77312
          }
        },
        "lat_ns" : {
          "min" : 6372,
          "max" : 135744,
          "mean" : 19900.015943,
          "stddev" : 2965.963506,
          "N" : 2773113
        },
        "bw_min" : 197423,
        "bw_max" : 198917,
        "bw_agg" : 100.000000,
        "bw_mean" : 198149.500000,
        "bw_dev" : 471.310897,
        "bw_samples" : 14,
        "iops_min" : 394847,
        "iops_max" : 397834,
        "iops_mean" : 396299.142857,
        "iops_stddev" : 942.487532,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 20.971429,
      "sys_cpu" : 76.928571,
      "ctx" : 16494,
      "majf" : 0,
      "minf" : 1,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 100.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 99.999982,
        "8" : 0.100000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.102989,
        "20" : 95.387482,
        "50" : 4.409135,
        "100" : 0.099780,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 8,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring-xnvme: (groupid=0, jobs=1): err= 0: pid=11022: Wed Oct 26 15:27:36 2022
  read: IOPS=396k, BW=193MiB/s (203MB/s)(1354MiB/7001msec)
    slat (nsec): min=1393, max=70382, avg=2032.97, stdev=315.23
    clat (usec): min=4, max=134, avg=17.87, stdev= 3.01
     lat (usec): min=6, max=135, avg=19.90, stdev= 2.97
    clat percentiles (nsec):
     |  1.00th=[13376],  5.00th=[15424], 10.00th=[16768], 20.00th=[17024],
     | 30.00th=[17280], 40.00th=[17536], 50.00th=[17536], 60.00th=[17536],
     | 70.00th=[18048], 80.00th=[18048], 90.00th=[18304], 95.00th=[19584],
     | 99.00th=[31616], 99.50th=[34560], 99.90th=[50432], 99.95th=[64256],
     | 99.99th=[77312]
   bw (  KiB/s): min=197423, max=198917, per=100.00%, avg=198149.50, stdev=471.31, samples=14
   iops        : min=394847, max=397834, avg=396299.14, stdev=942.49, samples=14
  lat (usec)   : 10=0.10%, 20=95.39%, 50=4.41%, 100=0.10%, 250=0.01%
  cpu          : usr=20.97%, sys=76.93%, ctx=16494, majf=0, minf=1
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=2773106,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
   READ: bw=193MiB/s (203MB/s), 193MiB/s-193MiB/s (203MB/s-203MB/s), io=1354MiB (1420MB), run=7001-7001msec

$ rm /tmp/fio-output_512_8_io_uring-xnvme_1.txt
rm: cannot remove '/tmp/fio-output_512_8_io_uring-xnvme_1.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring-xnvme" --rw="randread" --size="12G" --iodepth="8" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_8_io_uring-xnvme_1.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_8_io_uring-xnvme_1.txt
io_uring-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666790873,
  "timestamp_ms" : 1666790873848,
  "time" : "Wed Oct 26 15:27:53 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "8",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
        "xnvme_be" : "linux",
        "xnvme_async" : "io_uring",
        "xnvme_sync" : "nvme",
        "xnvme_admin" : "nvme",
        "xnvme_dev_nsid" : "1",
        "filename" : "/dev/nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 1414161408,
        "io_kbytes" : 1381017,
        "bw_bytes" : 201994201,
        "bw" : 197259,
        "iops" : 394518.925868,
        "runtime" : 7001,
        "total_ios" : 2762027,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1392,
          "max" : 30357,
          "mean" : 2037.367052,
          "stddev" : 294.648162,
          "N" : 2762027
        },
        "clat_ns" : {
          "min" : 4098,
          "max" : 135283,
          "mean" : 17936.435806,
          "stddev" : 3001.136611,
          "N" : 2762034,
          "percentile" : {
            "1.000000" : 13376,
            "5.000000" : 15552,
            "10.000000" : 16768,
            "20.000000" : 17280,
            "30.000000" : 17536,
            "40.000000" : 17536,
            "50.000000" : 17536,
            "60.000000" : 17536,
            "70.000000" : 18048,
            "80.000000" : 18304,
            "90.000000" : 18304,
            "95.000000" : 19584,
            "99.000000" : 31616,
            "99.500000" : 35072,
            "99.900000" : 51456,
            "99.950000" : 66048,
            "99.990000" : 76288
          }
        },
        "lat_ns" : {
          "min" : 7033,
          "max" : 136786,
          "mean" : 19973.801927,
          "stddev" : 2955.591710,
          "N" : 2762034
        },
        "bw_min" : 196032,
        "bw_max" : 198923,
        "bw_agg" : 100.000000,
        "bw_mean" : 197370.428571,
        "bw_dev" : 916.608608,
        "bw_samples" : 14,
        "iops_min" : 392064,
        "iops_max" : 397846,
        "iops_mean" : 394741.000000,
        "iops_stddev" : 1833.052687,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 20.842857,
      "sys_cpu" : 77.085714,
      "ctx" : 16725,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 100.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 99.999982,
        "8" : 0.100000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.081208,
        "20" : 95.414129,
        "50" : 4.391956,
        "100" : 0.112671,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 8,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring-xnvme: (groupid=0, jobs=1): err= 0: pid=11070: Wed Oct 26 15:27:53 2022
  read: IOPS=395k, BW=193MiB/s (202MB/s)(1349MiB/7001msec)
    slat (nsec): min=1392, max=30357, avg=2037.37, stdev=294.65
    clat (usec): min=4, max=135, avg=17.94, stdev= 3.00
     lat (usec): min=7, max=136, avg=19.97, stdev= 2.96
    clat percentiles (nsec):
     |  1.00th=[13376],  5.00th=[15552], 10.00th=[16768], 20.00th=[17280],
     | 30.00th=[17536], 40.00th=[17536], 50.00th=[17536], 60.00th=[17536],
     | 70.00th=[18048], 80.00th=[18304], 90.00th=[18304], 95.00th=[19584],
     | 99.00th=[31616], 99.50th=[35072], 99.90th=[51456], 99.95th=[66048],
     | 99.99th=[76288]
   bw (  KiB/s): min=196032, max=198923, per=100.00%, avg=197370.43, stdev=916.61, samples=14
   iops        : min=392064, max=397846, avg=394741.00, stdev=1833.05, samples=14
  lat (usec)   : 10=0.08%, 20=95.41%, 50=4.39%, 100=0.11%, 250=0.01%
  cpu          : usr=20.84%, sys=77.09%, ctx=16725, majf=0, minf=0
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=2762027,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
   READ: bw=193MiB/s (202MB/s), 193MiB/s-193MiB/s (202MB/s-202MB/s), io=1349MiB (1414MB), run=7001-7001msec

$ rm /tmp/fio-output_512_8_io_uring-xnvme_2.txt
rm: cannot remove '/tmp/fio-output_512_8_io_uring-xnvme_2.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring-xnvme" --rw="randread" --size="12G" --iodepth="8" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_8_io_uring-xnvme_2.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_1_io_uring-reference_2.txt
io_uring-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=io_uring, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666789304,
  "timestamp_ms" : 1666789304886,
  "time" : "Wed Oct 26 15:01:44 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-reference",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-reference",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "1",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "io_uring",
        "filename" : "/dev/nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 393098240,
        "io_kbytes" : 383885,
        "bw_bytes" : 56148870,
        "bw" : 54832,
        "iops" : 109665.762034,
        "runtime" : 7001,
        "total_ios" : 767770,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1493,
          "max" : 6332,
          "mean" : 1588.558791,
          "stddev" : 74.080772,
          "N" : 767770
        },
        "clat_ns" : {
          "min" : 2945,
          "max" : 114725,
          "mean" : 7224.469301,
          "stddev" : 999.991459,
          "N" : 767770,
          "percentile" : {
            "1.000000" : 6752,
            "5.000000" : 6944,
            "10.000000" : 6944,
            "20.000000" : 7008,
            "30.000000" : 7008,
            "40.000000" : 7072,
            "50.000000" : 7136,
            "60.000000" : 7136,
            "70.000000" : 7200,
            "80.000000" : 7200,
            "90.000000" : 7328,
            "95.000000" : 7456,
            "99.000000" : 10432,
            "99.500000" : 13632,
            "99.900000" : 15552,
            "99.950000" : 15808,
            "99.990000" : 59136
          }
        },
        "lat_ns" : {
          "min" : 7434,
          "max" : 116268,
          "mean" : 8813.028092,
          "stddev" : 1004.849320,
          "N" : 767770
        },
        "bw_min" : 54684,
        "bw_max" : 55007,
        "bw_agg" : 100.000000,
        "bw_mean" : 54870.642857,
        "bw_dev" : 100.171090,
        "bw_samples" : 14,
        "iops_min" : 109368,
        "iops_max" : 110014,
        "iops_mean" : 109741.500000,
        "iops_stddev" : 200.451701,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 7.971429,
      "sys_cpu" : 35.742857,
      "ctx" : 767772,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 100.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010000,
        "10" : 98.707556,
        "20" : 1.272517,
        "50" : 0.010000,
        "100" : 0.014588,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 1,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ],
  "disk_util" : [
    {
      "name" : "nvme0n1",
      "read_ios" : 1100470,
      "write_ios" : 0,
      "read_merges" : 0,
      "write_merges" : 0,
      "read_ticks" : 5516,
      "write_ticks" : 0,
      "in_queue" : 5516,
      "util" : 99.108381
    }
  ]
}

io_uring-reference: (groupid=0, jobs=1): err= 0: pid=5483: Wed Oct 26 15:01:44 2022
  read: IOPS=110k, BW=53.5MiB/s (56.1MB/s)(375MiB/7001msec)
    slat (nsec): min=1493, max=6332, avg=1588.56, stdev=74.08
    clat (nsec): min=2945, max=114725, avg=7224.47, stdev=999.99
     lat (usec): min=7, max=116, avg= 8.81, stdev= 1.00
    clat percentiles (nsec):
     |  1.00th=[ 6752],  5.00th=[ 6944], 10.00th=[ 6944], 20.00th=[ 7008],
     | 30.00th=[ 7008], 40.00th=[ 7072], 50.00th=[ 7136], 60.00th=[ 7136],
     | 70.00th=[ 7200], 80.00th=[ 7200], 90.00th=[ 7328], 95.00th=[ 7456],
     | 99.00th=[10432], 99.50th=[13632], 99.90th=[15552], 99.95th=[15808],
     | 99.99th=[59136]
   bw (  KiB/s): min=54684, max=55007, per=100.00%, avg=54870.64, stdev=100.17, samples=14
   iops        : min=109368, max=110014, avg=109741.50, stdev=200.45, samples=14
  lat (usec)   : 4=0.01%, 10=98.71%, 20=1.27%, 50=0.01%, 100=0.01%
  lat (usec)   : 250=0.01%
  cpu          : usr=7.97%, sys=35.74%, ctx=767772, majf=0, minf=0
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=767770,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=53.5MiB/s (56.1MB/s), 53.5MiB/s-53.5MiB/s (56.1MB/s-56.1MB/s), io=375MiB (393MB), run=7001-7001msec

Disk stats (read/write):
  nvme0n1: ios=1100470/0, merge=0/0, ticks=5516/0, in_queue=5516, util=99.11%

$ cat /tmp/fio-output_512_8_io_uring-xnvme_2.txt
io_uring-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666790891,
  "timestamp_ms" : 1666790891701,
  "time" : "Wed Oct 26 15:28:11 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "8",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
        "xnvme_be" : "linux",
        "xnvme_async" : "io_uring",
        "xnvme_sync" : "nvme",
        "xnvme_admin" : "nvme",
        "xnvme_dev_nsid" : "1",
        "filename" : "/dev/nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 1422672384,
        "io_kbytes" : 1389328,
        "bw_bytes" : 203209882,
        "bw" : 198447,
        "iops" : 396893.158120,
        "runtime" : 7001,
        "total_ios" : 2778649,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1402,
          "max" : 29996,
          "mean" : 2024.455626,
          "stddev" : 305.377303,
          "N" : 2778650
        },
        "clat_ns" : {
          "min" : 2254,
          "max" : 131687,
          "mean" : 17831.837688,
          "stddev" : 3036.634295,
          "N" : 2778656,
          "percentile" : {
            "1.000000" : 13248,
            "5.000000" : 15168,
            "10.000000" : 16512,
            "20.000000" : 17024,
            "30.000000" : 17280,
            "40.000000" : 17536,
            "50.000000" : 17536,
            "60.000000" : 17536,
            "70.000000" : 17536,
            "80.000000" : 18048,
            "90.000000" : 18304,
            "95.000000" : 19584,
            "99.000000" : 31872,
            "99.500000" : 35072,
            "99.900000" : 50944,
            "99.950000" : 64768,
            "99.990000" : 77312
          }
        },
        "lat_ns" : {
          "min" : 6422,
          "max" : 133160,
          "mean" : 19856.292749,
          "stddev" : 2987.817199,
          "N" : 2778656
        },
        "bw_min" : 197185,
        "bw_max" : 199877,
        "bw_agg" : 100.000000,
        "bw_mean" : 198482.571429,
        "bw_dev" : 752.010404,
        "bw_samples" : 14,
        "iops_min" : 394370,
        "iops_max" : 399754,
        "iops_mean" : 396965.142857,
        "iops_stddev" : 1504.020809,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 22.071429,
      "sys_cpu" : 75.800000,
      "ctx" : 16640,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 100.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 99.999982,
        "8" : 0.100000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010000,
        "10" : 0.090620,
        "20" : 95.460888,
        "50" : 4.343226,
        "100" : 0.104331,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 8,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring-xnvme: (groupid=0, jobs=1): err= 0: pid=11118: Wed Oct 26 15:28:11 2022
  read: IOPS=397k, BW=194MiB/s (203MB/s)(1357MiB/7001msec)
    slat (nsec): min=1402, max=29996, avg=2024.46, stdev=305.38
    clat (usec): min=2, max=131, avg=17.83, stdev= 3.04
     lat (usec): min=6, max=133, avg=19.86, stdev= 2.99
    clat percentiles (nsec):
     |  1.00th=[13248],  5.00th=[15168], 10.00th=[16512], 20.00th=[17024],
     | 30.00th=[17280], 40.00th=[17536], 50.00th=[17536], 60.00th=[17536],
     | 70.00th=[17536], 80.00th=[18048], 90.00th=[18304], 95.00th=[19584],
     | 99.00th=[31872], 99.50th=[35072], 99.90th=[50944], 99.95th=[64768],
     | 99.99th=[77312]
   bw (  KiB/s): min=197185, max=199877, per=100.00%, avg=198482.57, stdev=752.01, samples=14
   iops        : min=394370, max=399754, avg=396965.14, stdev=1504.02, samples=14
  lat (usec)   : 4=0.01%, 10=0.09%, 20=95.46%, 50=4.34%, 100=0.10%
  lat (usec)   : 250=0.01%
  cpu          : usr=22.07%, sys=75.80%, ctx=16640, majf=0, minf=0
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=2778649,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
   READ: bw=194MiB/s (203MB/s), 194MiB/s-194MiB/s (203MB/s-203MB/s), io=1357MiB (1423MB), run=7001-7001msec

$ rm /tmp/fio-output_512_8_libaio-reference_0.txt
rm: cannot remove '/tmp/fio-output_512_8_libaio-reference_0.txt': No such file or directory

$ /opt/fio/bin/fio --name="libaio-reference" --rw="randread" --size="12G" --iodepth="8" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="libaio" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_8_libaio-reference_0.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_8_libaio-reference_0.txt
libaio-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=libaio, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666790907,
  "timestamp_ms" : 1666790907820,
  "time" : "Wed Oct 26 15:28:27 2022",
  "jobs" : [
    {
      "jobname" : "libaio-reference",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-reference",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "8",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "libaio",
        "filename" : "/dev/nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 1573419008,
        "io_kbytes" : 1536542,
        "bw_bytes" : 224742037,
        "bw" : 219474,
        "iops" : 438948.293101,
        "runtime" : 7001,
        "total_ios" : 3073077,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1442,
          "max" : 20178,
          "mean" : 1602.113093,
          "stddev" : 122.876606,
          "N" : 3073077
        },
        "clat_ns" : {
          "min" : 4669,
          "max" : 139171,
          "mean" : 16380.314330,
          "stddev" : 3042.824860,
          "N" : 3073084,
          "percentile" : {
            "1.000000" : 13888,
            "5.000000" : 15552,
            "10.000000" : 15680,
            "20.000000" : 15680,
            "30.000000" : 15680,
            "40.000000" : 15808,
            "50.000000" : 15808,
            "60.000000" : 15808,
            "70.000000" : 15936,
            "80.000000" : 16320,
            "90.000000" : 16512,
            "95.000000" : 17536,
            "99.000000" : 31616,
            "99.500000" : 35584,
            "99.900000" : 51456,
            "99.950000" : 66048,
            "99.990000" : 78336
          }
        },
        "lat_ns" : {
          "min" : 6172,
          "max" : 140724,
          "mean" : 17982.427350,
          "stddev" : 3044.762520,
          "N" : 3073084
        },
        "bw_min" : 218423,
        "bw_max" : 220928,
        "bw_agg" : 100.000000,
        "bw_mean" : 219536.928571,
        "bw_dev" : 658.497411,
        "bw_samples" : 14,
        "iops_min" : 436846,
        "iops_max" : 441857,
        "iops_mean" : 439073.928571,
        "iops_stddev" : 1317.157339,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 30.700000,
      "sys_cpu" : 67.314286,
      "ctx" : 18155,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 100.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 99.999967,
        "8" : 0.100000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.426283,
        "20" : 95.925061,
        "50" : 3.544721,
        "100" : 0.103024,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 8,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ],
  "disk_util" : [
    {
      "name" : "nvme0n1",
      "read_ios" : 4442260,
      "write_ios" : 0,
      "read_merges" : 0,
      "write_merges" : 0,
      "read_ticks" : 25258,
      "write_ticks" : 0,
      "in_queue" : 25258,
      "util" : 99.069123
    }
  ]
}

libaio-reference: (groupid=0, jobs=1): err= 0: pid=11167: Wed Oct 26 15:28:27 2022
  read: IOPS=439k, BW=214MiB/s (225MB/s)(1501MiB/7001msec)
    slat (nsec): min=1442, max=20178, avg=1602.11, stdev=122.88
    clat (usec): min=4, max=139, avg=16.38, stdev= 3.04
     lat (usec): min=6, max=140, avg=17.98, stdev= 3.04
    clat percentiles (nsec):
     |  1.00th=[13888],  5.00th=[15552], 10.00th=[15680], 20.00th=[15680],
     | 30.00th=[15680], 40.00th=[15808], 50.00th=[15808], 60.00th=[15808],
     | 70.00th=[15936], 80.00th=[16320], 90.00th=[16512], 95.00th=[17536],
     | 99.00th=[31616], 99.50th=[35584], 99.90th=[51456], 99.95th=[66048],
     | 99.99th=[78336]
   bw (  KiB/s): min=218423, max=220928, per=100.00%, avg=219536.93, stdev=658.50, samples=14
   iops        : min=436846, max=441857, avg=439073.93, stdev=1317.16, samples=14
  lat (usec)   : 10=0.43%, 20=95.93%, 50=3.54%, 100=0.10%, 250=0.01%
  cpu          : usr=30.70%, sys=67.31%, ctx=18155, majf=0, minf=0
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=3073077,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
   READ: bw=214MiB/s (225MB/s), 214MiB/s-214MiB/s (225MB/s-225MB/s), io=1501MiB (1573MB), run=7001-7001msec

Disk stats (read/write):
  nvme0n1: ios=4442260/0, merge=0/0, ticks=25258/0, in_queue=25258, util=99.07%

$ rm /tmp/fio-output_512_8_libaio-reference_1.txt
rm: cannot remove '/tmp/fio-output_512_8_libaio-reference_1.txt': No such file or directory

$ /opt/fio/bin/fio --name="libaio-reference" --rw="randread" --size="12G" --iodepth="8" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="libaio" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_8_libaio-reference_1.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_8_libaio-reference_1.txt
libaio-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=libaio, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666790924,
  "timestamp_ms" : 1666790924175,
  "time" : "Wed Oct 26 15:28:44 2022",
  "jobs" : [
    {
      "jobname" : "libaio-reference",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-reference",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "8",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "libaio",
        "filename" : "/dev/nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 1567867904,
        "io_kbytes" : 1531121,
        "bw_bytes" : 223949136,
        "bw" : 218700,
        "iops" : 437399.514355,
        "runtime" : 7001,
        "total_ios" : 3062234,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1442,
          "max" : 80601,
          "mean" : 1606.787758,
          "stddev" : 104.510126,
          "N" : 3062235
        },
        "clat_ns" : {
          "min" : 4960,
          "max" : 135585,
          "mean" : 16440.666159,
          "stddev" : 2924.034150,
          "N" : 3062241,
          "percentile" : {
            "1.000000" : 13760,
            "5.000000" : 15680,
            "10.000000" : 15680,
            "20.000000" : 15808,
            "30.000000" : 15808,
            "40.000000" : 15808,
            "50.000000" : 15936,
            "60.000000" : 15936,
            "70.000000" : 16064,
            "80.000000" : 16512,
            "90.000000" : 16512,
            "95.000000" : 17536,
            "99.000000" : 31104,
            "99.500000" : 35072,
            "99.900000" : 48896,
            "99.950000" : 63744,
            "99.990000" : 78336
          }
        },
        "lat_ns" : {
          "min" : 6483,
          "max" : 137268,
          "mean" : 18047.453991,
          "stddev" : 2925.765710,
          "N" : 3062241
        },
        "bw_min" : 218165,
        "bw_max" : 219778,
        "bw_agg" : 100.000000,
        "bw_mean" : 218823.571429,
        "bw_dev" : 414.416868,
        "bw_samples" : 14,
        "iops_min" : 436330,
        "iops_max" : 439557,
        "iops_mean" : 437647.285714,
        "iops_stddev" : 829.026624,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 31.600000,
      "sys_cpu" : 66.528571,
      "ctx" : 17763,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 100.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 99.999967,
        "8" : 0.100000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.421882,
        "20" : 96.022054,
        "50" : 3.461231,
        "100" : 0.094180,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 8,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ],
  "disk_util" : [
    {
      "name" : "nvme0n1",
      "read_ios" : 4419891,
      "write_ios" : 0,
      "read_merges" : 0,
      "write_merges" : 0,
      "read_ticks" : 25004,
      "write_ticks" : 0,
      "in_queue" : 25004,
      "util" : 99.088748
    }
  ]
}

libaio-reference: (groupid=0, jobs=1): err= 0: pid=11213: Wed Oct 26 15:28:44 2022
  read: IOPS=437k, BW=214MiB/s (224MB/s)(1495MiB/7001msec)
    slat (nsec): min=1442, max=80601, avg=1606.79, stdev=104.51
    clat (usec): min=4, max=135, avg=16.44, stdev= 2.92
     lat (usec): min=6, max=137, avg=18.05, stdev= 2.93
    clat percentiles (nsec):
     |  1.00th=[13760],  5.00th=[15680], 10.00th=[15680], 20.00th=[15808],
     | 30.00th=[15808], 40.00th=[15808], 50.00th=[15936], 60.00th=[15936],
     | 70.00th=[16064], 80.00th=[16512], 90.00th=[16512], 95.00th=[17536],
     | 99.00th=[31104], 99.50th=[35072], 99.90th=[48896], 99.95th=[63744],
     | 99.99th=[78336]
   bw (  KiB/s): min=218165, max=219778, per=100.00%, avg=218823.57, stdev=414.42, samples=14
   iops        : min=436330, max=439557, avg=437647.29, stdev=829.03, samples=14
  lat (usec)   : 10=0.42%, 20=96.02%, 50=3.46%, 100=0.09%, 250=0.01%
  cpu          : usr=31.60%, sys=66.53%, ctx=17763, majf=0, minf=0
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=3062234,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
   READ: bw=214MiB/s (224MB/s), 214MiB/s-214MiB/s (224MB/s-224MB/s), io=1495MiB (1568MB), run=7001-7001msec

Disk stats (read/write):
  nvme0n1: ios=4419891/0, merge=0/0, ticks=25004/0, in_queue=25004, util=99.09%

$ rm /tmp/fio-output_512_8_libaio-reference_2.txt
rm: cannot remove '/tmp/fio-output_512_8_libaio-reference_2.txt': No such file or directory

$ /opt/fio/bin/fio --name="libaio-reference" --rw="randread" --size="12G" --iodepth="8" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="libaio" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_8_libaio-reference_2.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_8_libaio-reference_2.txt
libaio-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=libaio, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666790941,
  "timestamp_ms" : 1666790941607,
  "time" : "Wed Oct 26 15:29:01 2022",
  "jobs" : [
    {
      "jobname" : "libaio-reference",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-reference",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "8",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "libaio",
        "filename" : "/dev/nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 1575713792,
        "io_kbytes" : 1538783,
        "bw_bytes" : 225069817,
        "bw" : 219794,
        "iops" : 439588.344522,
        "runtime" : 7001,
        "total_ios" : 3077558,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1442,
          "max" : 20608,
          "mean" : 1603.968562,
          "stddev" : 112.982422,
          "N" : 3077559
        },
        "clat_ns" : {
          "min" : 4729,
          "max" : 131707,
          "mean" : 16357.507495,
          "stddev" : 3033.941500,
          "N" : 3077565,
          "percentile" : {
            "1.000000" : 13632,
            "5.000000" : 15552,
            "10.000000" : 15552,
            "20.000000" : 15680,
            "30.000000" : 15680,
            "40.000000" : 15808,
            "50.000000" : 15808,
            "60.000000" : 15808,
            "70.000000" : 15936,
            "80.000000" : 16320,
            "90.000000" : 16512,
            "95.000000" : 17536,
            "99.000000" : 31872,
            "99.500000" : 36096,
            "99.900000" : 49408,
            "99.950000" : 64256,
            "99.990000" : 77312
          }
        },
        "lat_ns" : {
          "min" : 6242,
          "max" : 133490,
          "mean" : 17961.475945,
          "stddev" : 3035.114182,
          "N" : 3077565
        },
        "bw_min" : 219076,
        "bw_max" : 222226,
        "bw_agg" : 100.000000,
        "bw_mean" : 219857.071429,
        "bw_dev" : 825.530735,
        "bw_samples" : 14,
        "iops_min" : 438152,
        "iops_max" : 444452,
        "iops_mean" : 439714.285714,
        "iops_stddev" : 1651.104219,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 30.700000,
      "sys_cpu" : 67.271429,
      "ctx" : 18039,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 100.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 99.999968,
        "8" : 0.100000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.440284,
        "20" : 95.900321,
        "50" : 3.560713,
        "100" : 0.098650,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 8,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ],
  "disk_util" : [
    {
      "name" : "nvme0n1",
      "read_ios" : 4443543,
      "write_ios" : 0,
      "read_merges" : 0,
      "write_merges" : 0,
      "read_ticks" : 25340,
      "write_ticks" : 0,
      "in_queue" : 25340,
      "util" : 99.069123
    }
  ]
}

libaio-reference: (groupid=0, jobs=1): err= 0: pid=11259: Wed Oct 26 15:29:01 2022
  read: IOPS=440k, BW=215MiB/s (225MB/s)(1503MiB/7001msec)
    slat (nsec): min=1442, max=20608, avg=1603.97, stdev=112.98
    clat (usec): min=4, max=131, avg=16.36, stdev= 3.03
     lat (usec): min=6, max=133, avg=17.96, stdev= 3.04
    clat percentiles (nsec):
     |  1.00th=[13632],  5.00th=[15552], 10.00th=[15552], 20.00th=[15680],
     | 30.00th=[15680], 40.00th=[15808], 50.00th=[15808], 60.00th=[15808],
     | 70.00th=[15936], 80.00th=[16320], 90.00th=[16512], 95.00th=[17536],
     | 99.00th=[31872], 99.50th=[36096], 99.90th=[49408], 99.95th=[64256],
     | 99.99th=[77312]
   bw (  KiB/s): min=219076, max=222226, per=100.00%, avg=219857.07, stdev=825.53, samples=14
   iops        : min=438152, max=444452, avg=439714.29, stdev=1651.10, samples=14
  lat (usec)   : 10=0.44%, 20=95.90%, 50=3.56%, 100=0.10%, 250=0.01%
  cpu          : usr=30.70%, sys=67.27%, ctx=18039, majf=0, minf=0
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=3077558,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
   READ: bw=215MiB/s (225MB/s), 215MiB/s-215MiB/s (225MB/s-225MB/s), io=1503MiB (1576MB), run=7001-7001msec

Disk stats (read/write):
  nvme0n1: ios=4443543/0, merge=0/0, ticks=25340/0, in_queue=25340, util=99.07%

$ rm /tmp/fio-output_512_1_io_uring-bdev_uring_0.txt
rm: cannot remove '/tmp/fio-output_512_1_io_uring-bdev_uring_0.txt': No such file or directory

$ rm /tmp/fio-output_512_8_libaio-bdev_aio_0.txt
rm: cannot remove '/tmp/fio-output_512_8_libaio-bdev_aio_0.txt': No such file or directory

$ /opt/fio/bin/fio --name="libaio-bdev_aio" --rw="randread" --size="12G" --iodepth="8" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_aio_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_8_libaio-bdev_aio_0.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_512_8_libaio-bdev_aio_0.txt
libaio-bdev_aio: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666790960,
  "timestamp_ms" : 1666790960594,
  "time" : "Wed Oct 26 15:29:20 2022",
  "jobs" : [
    {
      "jobname" : "libaio-bdev_aio",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-bdev_aio",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "8",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_aio_foo.conf",
        "filename" : "Nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 1661046784,
        "io_kbytes" : 1622116,
        "bw_bytes" : 237258503,
        "bw" : 231697,
        "iops" : 463394.372233,
        "runtime" : 7001,
        "total_ios" : 3244224,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1633,
          "max" : 22713,
          "mean" : 1818.265835,
          "stddev" : 123.789028,
          "N" : 3244225
        },
        "clat_ns" : {
          "min" : 361,
          "max" : 109205,
          "mean" : 10578.592328,
          "stddev" : 4535.960378,
          "N" : 3244224,
          "percentile" : {
            "1.000000" : 4192,
            "5.000000" : 4320,
            "10.000000" : 6112,
            "20.000000" : 6304,
            "30.000000" : 8160,
            "40.000000" : 8384,
            "50.000000" : 10432,
            "60.000000" : 11200,
            "70.000000" : 12864,
            "80.000000" : 14528,
            "90.000000" : 14784,
            "95.000000" : 15296,
            "99.000000" : 26496,
            "99.500000" : 28288,
            "99.900000" : 43776,
            "99.950000" : 60160,
            "99.990000" : 72192
          }
        },
        "lat_ns" : {
          "min" : 5740,
          "max" : 111099,
          "mean" : 12396.858125,
          "stddev" : 4531.767359,
          "N" : 3244224
        },
        "bw_min" : 230450,
        "bw_max" : 233465,
        "bw_agg" : 100.000000,
        "bw_mean" : 231740.142857,
        "bw_dev" : 820.889841,
        "bw_samples" : 14,
        "iops_min" : 460900,
        "iops_max" : 466930,
        "iops_mean" : 463480.285714,
        "iops_stddev" : 1641.459736,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 36.971429,
      "sys_cpu" : 63.014286,
      "ctx" : 5,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.100000,
        "2" : 8.133347,
        "4" : 72.486425,
        "8" : 19.379550,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 85.202722,
        "8" : 14.797278,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.010000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010000,
        "10" : 43.187308,
        "20" : 54.000433,
        "50" : 2.732826,
        "100" : 0.077307,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 8,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

libaio-bdev_aio: (groupid=0, jobs=1): err= 0: pid=11316: Wed Oct 26 15:29:20 2022
  read: IOPS=463k, BW=226MiB/s (237MB/s)(1584MiB/7001msec)
    slat (nsec): min=1633, max=22713, avg=1818.27, stdev=123.79
    clat (nsec): min=361, max=109205, avg=10578.59, stdev=4535.96
     lat (usec): min=5, max=111, avg=12.40, stdev= 4.53
    clat percentiles (nsec):
     |  1.00th=[ 4192],  5.00th=[ 4320], 10.00th=[ 6112], 20.00th=[ 6304],
     | 30.00th=[ 8160], 40.00th=[ 8384], 50.00th=[10432], 60.00th=[11200],
     | 70.00th=[12864], 80.00th=[14528], 90.00th=[14784], 95.00th=[15296],
     | 99.00th=[26496], 99.50th=[28288], 99.90th=[43776], 99.95th=[60160],
     | 99.99th=[72192]
   bw (  KiB/s): min=230450, max=233465, per=100.00%, avg=231740.14, stdev=820.89, samples=14
   iops        : min=460900, max=466930, avg=463480.29, stdev=1641.46, samples=14
  lat (nsec)   : 500=0.01%
  lat (usec)   : 4=0.01%, 10=43.19%, 20=54.00%, 50=2.73%, 100=0.08%
  lat (usec)   : 250=0.01%
  cpu          : usr=36.97%, sys=63.01%, ctx=5, majf=0, minf=0
  IO depths    : 1=0.1%, 2=8.1%, 4=72.5%, 8=19.4%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=85.2%, 8=14.8%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=3244224,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
   READ: bw=226MiB/s (237MB/s), 226MiB/s-226MiB/s (237MB/s-237MB/s), io=1584MiB (1661MB), run=7001-7001msec

$ rm /tmp/fio-output_512_8_libaio-bdev_aio_1.txt
rm: cannot remove '/tmp/fio-output_512_8_libaio-bdev_aio_1.txt': No such file or directory

$ /opt/fio/bin/fio --name="libaio-bdev_aio" --rw="randread" --size="12G" --iodepth="8" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_aio_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_8_libaio-bdev_aio_1.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_512_8_libaio-bdev_aio_1.txt
libaio-bdev_aio: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666790977,
  "timestamp_ms" : 1666790977538,
  "time" : "Wed Oct 26 15:29:37 2022",
  "jobs" : [
    {
      "jobname" : "libaio-bdev_aio",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-bdev_aio",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "8",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_aio_foo.conf",
        "filename" : "Nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 1666486272,
        "io_kbytes" : 1627428,
        "bw_bytes" : 238035462,
        "bw" : 232456,
        "iops" : 464912.012570,
        "runtime" : 7001,
        "total_ios" : 3254849,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1623,
          "max" : 23605,
          "mean" : 1803.010349,
          "stddev" : 126.458209,
          "N" : 3254850
        },
        "clat_ns" : {
          "min" : 370,
          "max" : 125115,
          "mean" : 10935.840299,
          "stddev" : 4372.129142,
          "N" : 3254856,
          "percentile" : {
            "1.000000" : 4256,
            "5.000000" : 6112,
            "10.000000" : 6176,
            "20.000000" : 6560,
            "30.000000" : 8160,
            "40.000000" : 10048,
            "50.000000" : 10688,
            "60.000000" : 12480,
            "70.000000" : 12736,
            "80.000000" : 14400,
            "90.000000" : 14656,
            "95.000000" : 15168,
            "99.000000" : 27008,
            "99.500000" : 28800,
            "99.900000" : 43264,
            "99.950000" : 60672,
            "99.990000" : 72192
          }
        },
        "lat_ns" : {
          "min" : 5661,
          "max" : 126928,
          "mean" : 12738.850612,
          "stddev" : 4363.905228,
          "N" : 3254856
        },
        "bw_min" : 229362,
        "bw_max" : 234011,
        "bw_agg" : 100.000000,
        "bw_mean" : 232498.785714,
        "bw_dev" : 1354.748752,
        "bw_samples" : 14,
        "iops_min" : 458724,
        "iops_max" : 468022,
        "iops_mean" : 464997.642857,
        "iops_stddev" : 2709.526147,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 41.585714,
      "sys_cpu" : 58.385714,
      "ctx" : 7,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.100000,
        "2" : 1.864664,
        "4" : 77.562246,
        "8" : 20.571922,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 84.411387,
        "8" : 15.588613,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.010000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010000,
        "10" : 39.686818,
        "20" : 57.371356,
        "50" : 2.863297,
        "100" : 0.075702,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 8,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

libaio-bdev_aio: (groupid=0, jobs=1): err= 0: pid=11373: Wed Oct 26 15:29:37 2022
  read: IOPS=465k, BW=227MiB/s (238MB/s)(1589MiB/7001msec)
    slat (nsec): min=1623, max=23605, avg=1803.01, stdev=126.46
    clat (nsec): min=370, max=125115, avg=10935.84, stdev=4372.13
     lat (usec): min=5, max=126, avg=12.74, stdev= 4.36
    clat percentiles (nsec):
     |  1.00th=[ 4256],  5.00th=[ 6112], 10.00th=[ 6176], 20.00th=[ 6560],
     | 30.00th=[ 8160], 40.00th=[10048], 50.00th=[10688], 60.00th=[12480],
     | 70.00th=[12736], 80.00th=[14400], 90.00th=[14656], 95.00th=[15168],
     | 99.00th=[27008], 99.50th=[28800], 99.90th=[43264], 99.95th=[60672],
     | 99.99th=[72192]
   bw (  KiB/s): min=229362, max=234011, per=100.00%, avg=232498.79, stdev=1354.75, samples=14
   iops        : min=458724, max=468022, avg=464997.64, stdev=2709.53, samples=14
  lat (nsec)   : 500=0.01%
  lat (usec)   : 4=0.01%, 10=39.69%, 20=57.37%, 50=2.86%, 100=0.08%
  lat (usec)   : 250=0.01%
  cpu          : usr=41.59%, sys=58.39%, ctx=7, majf=0, minf=0
  IO depths    : 1=0.1%, 2=1.9%, 4=77.6%, 8=20.6%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=84.4%, 8=15.6%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=3254849,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
   READ: bw=227MiB/s (238MB/s), 227MiB/s-227MiB/s (238MB/s-238MB/s), io=1589MiB (1666MB), run=7001-7001msec

$ rm /tmp/fio-output_512_8_libaio-bdev_aio_2.txt
rm: cannot remove '/tmp/fio-output_512_8_libaio-bdev_aio_2.txt': No such file or directory

$ /opt/fio/bin/fio --name="libaio-bdev_aio" --rw="randread" --size="12G" --iodepth="8" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_aio_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_8_libaio-bdev_aio_2.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_512_8_libaio-bdev_aio_2.txt
libaio-bdev_aio: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666790992,
  "timestamp_ms" : 1666790992810,
  "time" : "Wed Oct 26 15:29:52 2022",
  "jobs" : [
    {
      "jobname" : "libaio-bdev_aio",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-bdev_aio",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "8",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_aio_foo.conf",
        "filename" : "Nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 1667806208,
        "io_kbytes" : 1628717,
        "bw_bytes" : 238223997,
        "bw" : 232640,
        "iops" : 465280.531353,
        "runtime" : 7001,
        "total_ios" : 3257429,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1623,
          "max" : 23795,
          "mean" : 1807.492342,
          "stddev" : 121.710490,
          "N" : 3257430
        },
        "clat_ns" : {
          "min" : 391,
          "max" : 110808,
          "mean" : 10528.263130,
          "stddev" : 4533.572349,
          "N" : 3257434,
          "percentile" : {
            "1.000000" : 4192,
            "5.000000" : 4256,
            "10.000000" : 6112,
            "20.000000" : 6240,
            "30.000000" : 8096,
            "40.000000" : 8256,
            "50.000000" : 10304,
            "60.000000" : 11200,
            "70.000000" : 12736,
            "80.000000" : 14528,
            "90.000000" : 14784,
            "95.000000" : 15296,
            "99.000000" : 26240,
            "99.500000" : 28288,
            "99.900000" : 43264,
            "99.950000" : 60672,
            "99.990000" : 72192
          }
        },
        "lat_ns" : {
          "min" : 5650,
          "max" : 112531,
          "mean" : 12335.755434,
          "stddev" : 4529.016440,
          "N" : 3257434
        },
        "bw_min" : 230862,
        "bw_max" : 235114,
        "bw_agg" : 100.000000,
        "bw_mean" : 232693.000000,
        "bw_dev" : 1547.197367,
        "bw_samples" : 14,
        "iops_min" : 461716,
        "iops_max" : 470228,
        "iops_mean" : 465386.000000,
        "iops_stddev" : 3095.091101,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 38.057143,
      "sys_cpu" : 61.928571,
      "ctx" : 4,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.100000,
        "2" : 8.325922,
        "4" : 72.295359,
        "8" : 19.377736,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 85.258211,
        "8" : 14.741789,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.010000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010000,
        "10" : 43.861094,
        "20" : 53.338047,
        "50" : 2.724081,
        "100" : 0.074261,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 8,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

libaio-bdev_aio: (groupid=0, jobs=1): err= 0: pid=11431: Wed Oct 26 15:29:52 2022
  read: IOPS=465k, BW=227MiB/s (238MB/s)(1591MiB/7001msec)
    slat (nsec): min=1623, max=23795, avg=1807.49, stdev=121.71
    clat (nsec): min=391, max=110808, avg=10528.26, stdev=4533.57
     lat (usec): min=5, max=112, avg=12.34, stdev= 4.53
    clat percentiles (nsec):
     |  1.00th=[ 4192],  5.00th=[ 4256], 10.00th=[ 6112], 20.00th=[ 6240],
     | 30.00th=[ 8096], 40.00th=[ 8256], 50.00th=[10304], 60.00th=[11200],
     | 70.00th=[12736], 80.00th=[14528], 90.00th=[14784], 95.00th=[15296],
     | 99.00th=[26240], 99.50th=[28288], 99.90th=[43264], 99.95th=[60672],
     | 99.99th=[72192]
   bw (  KiB/s): min=230862, max=235114, per=100.00%, avg=232693.00, stdev=1547.20, samples=14
   iops        : min=461716, max=470228, avg=465386.00, stdev=3095.09, samples=14
  lat (nsec)   : 500=0.01%
  lat (usec)   : 4=0.01%, 10=43.86%, 20=53.34%, 50=2.72%, 100=0.07%
  lat (usec)   : 250=0.01%
  cpu          : usr=38.06%, sys=61.93%, ctx=4, majf=0, minf=0
  IO depths    : 1=0.1%, 2=8.3%, 4=72.3%, 8=19.4%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=85.3%, 8=14.7%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=3257429,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
   READ: bw=227MiB/s (238MB/s), 227MiB/s-227MiB/s (238MB/s-238MB/s), io=1591MiB (1668MB), run=7001-7001msec

$ rm /tmp/fio-output_512_8_libaio-bdev_xnvme_0.txt
rm: cannot remove '/tmp/fio-output_512_8_libaio-bdev_xnvme_0.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring-bdev_uring" --rw="randread" --size="12G" --iodepth="1" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_uring_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_1_io_uring-bdev_uring_0.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created

$ /opt/fio/bin/fio --name="libaio-bdev_xnvme" --rw="randread" --size="12G" --iodepth="8" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_8_libaio-bdev_xnvme_0.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_512_8_libaio-bdev_xnvme_0.txt
libaio-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666791008,
  "timestamp_ms" : 1666791008196,
  "time" : "Wed Oct 26 15:30:08 2022",
  "jobs" : [
    {
      "jobname" : "libaio-bdev_xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-bdev_xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "8",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_xnvme_libaio.conf",
        "filename" : "Nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 1567925248,
        "io_kbytes" : 1531177,
        "bw_bytes" : 223957327,
        "bw" : 218708,
        "iops" : 437415.940580,
        "runtime" : 7001,
        "total_ios" : 3062349,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1663,
          "max" : 34314,
          "mean" : 1858.695535,
          "stddev" : 122.078812,
          "N" : 3062350
        },
        "clat_ns" : {
          "min" : 831,
          "max" : 122730,
          "mean" : 11448.851285,
          "stddev" : 4591.776289,
          "N" : 3062354,
          "percentile" : {
            "1.000000" : 4704,
            "5.000000" : 4896,
            "10.000000" : 6688,
            "20.000000" : 6880,
            "30.000000" : 8768,
            "40.000000" : 9152,
            "50.000000" : 11584,
            "60.000000" : 12352,
            "70.000000" : 13888,
            "80.000000" : 15680,
            "90.000000" : 16064,
            "95.000000" : 16768,
            "99.000000" : 26240,
            "99.500000" : 29056,
            "99.900000" : 44800,
            "99.950000" : 59648,
            "99.990000" : 72192
          }
        },
        "lat_ns" : {
          "min" : 5661,
          "max" : 124534,
          "mean" : 13307.546791,
          "stddev" : 4584.897019,
          "N" : 3062354
        },
        "bw_min" : 218159,
        "bw_max" : 219551,
        "bw_agg" : 100.000000,
        "bw_mean" : 218814.142857,
        "bw_dev" : 384.929535,
        "bw_samples" : 14,
        "iops_min" : 436318,
        "iops_max" : 439102,
        "iops_mean" : 437628.428571,
        "iops_stddev" : 769.682623,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 39.185714,
      "sys_cpu" : 60.785714,
      "ctx" : 10,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.100000,
        "2" : 7.306744,
        "4" : 73.195772,
        "8" : 19.496700,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 85.097695,
        "8" : 14.902305,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.010000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010000,
        "10" : 42.621955,
        "20" : 54.869220,
        "50" : 2.423107,
        "100" : 0.082584,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 8,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

libaio-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=11488: Wed Oct 26 15:30:08 2022
  read: IOPS=437k, BW=214MiB/s (224MB/s)(1495MiB/7001msec)
    slat (nsec): min=1663, max=34314, avg=1858.70, stdev=122.08
    clat (nsec): min=831, max=122730, avg=11448.85, stdev=4591.78
     lat (usec): min=5, max=124, avg=13.31, stdev= 4.58
    clat percentiles (nsec):
     |  1.00th=[ 4704],  5.00th=[ 4896], 10.00th=[ 6688], 20.00th=[ 6880],
     | 30.00th=[ 8768], 40.00th=[ 9152], 50.00th=[11584], 60.00th=[12352],
     | 70.00th=[13888], 80.00th=[15680], 90.00th=[16064], 95.00th=[16768],
     | 99.00th=[26240], 99.50th=[29056], 99.90th=[44800], 99.95th=[59648],
     | 99.99th=[72192]
   bw (  KiB/s): min=218159, max=219551, per=100.00%, avg=218814.14, stdev=384.93, samples=14
   iops        : min=436318, max=439102, avg=437628.43, stdev=769.68, samples=14
  lat (nsec)   : 1000=0.01%
  lat (usec)   : 4=0.01%, 10=42.62%, 20=54.87%, 50=2.42%, 100=0.08%
  lat (usec)   : 250=0.01%
  cpu          : usr=39.19%, sys=60.79%, ctx=10, majf=0, minf=0
  IO depths    : 1=0.1%, 2=7.3%, 4=73.2%, 8=19.5%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=85.1%, 8=14.9%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=3062349,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
   READ: bw=214MiB/s (224MB/s), 214MiB/s-214MiB/s (224MB/s-224MB/s), io=1495MiB (1568MB), run=7001-7001msec

$ rm /tmp/fio-output_512_8_libaio-bdev_xnvme_1.txt
rm: cannot remove '/tmp/fio-output_512_8_libaio-bdev_xnvme_1.txt': No such file or directory

$ /opt/fio/bin/fio --name="libaio-bdev_xnvme" --rw="randread" --size="12G" --iodepth="8" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_8_libaio-bdev_xnvme_1.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_512_8_libaio-bdev_xnvme_1.txt
libaio-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666791023,
  "timestamp_ms" : 1666791023513,
  "time" : "Wed Oct 26 15:30:23 2022",
  "jobs" : [
    {
      "jobname" : "libaio-bdev_xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-bdev_xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "8",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_xnvme_libaio.conf",
        "filename" : "Nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 1576878592,
        "io_kbytes" : 1539920,
        "bw_bytes" : 225236193,
        "bw" : 219957,
        "iops" : 439913.726610,
        "runtime" : 7001,
        "total_ios" : 3079836,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1663,
          "max" : 19526,
          "mean" : 1853.081294,
          "stddev" : 134.461061,
          "N" : 3079837
        },
        "clat_ns" : {
          "min" : 812,
          "max" : 111990,
          "mean" : 11416.888518,
          "stddev" : 4588.974480,
          "N" : 3079841,
          "percentile" : {
            "1.000000" : 4704,
            "5.000000" : 4896,
            "10.000000" : 6624,
            "20.000000" : 6816,
            "30.000000" : 8768,
            "40.000000" : 9152,
            "50.000000" : 11584,
            "60.000000" : 12352,
            "70.000000" : 13888,
            "80.000000" : 15552,
            "90.000000" : 15936,
            "95.000000" : 16512,
            "99.000000" : 26752,
            "99.500000" : 29312,
            "99.900000" : 44288,
            "99.950000" : 60160,
            "99.990000" : 73216
          }
        },
        "lat_ns" : {
          "min" : 5660,
          "max" : 113774,
          "mean" : 13269.969731,
          "stddev" : 4580.597825,
          "N" : 3079841
        },
        "bw_min" : 219083,
        "bw_max" : 221465,
        "bw_agg" : 100.000000,
        "bw_mean" : 219996.928571,
        "bw_dev" : 649.316737,
        "bw_samples" : 14,
        "iops_min" : 438166,
        "iops_max" : 442931,
        "iops_mean" : 439993.928571,
        "iops_stddev" : 1298.760028,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 37.185714,
      "sys_cpu" : 62.800000,
      "ctx" : 5,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.100000,
        "2" : 6.747048,
        "4" : 73.657980,
        "8" : 19.593803,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 85.015234,
        "8" : 14.984766,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.010000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010000,
        "10" : 42.298421,
        "20" : 55.114980,
        "50" : 2.504776,
        "100" : 0.078511,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 8,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

libaio-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=11545: Wed Oct 26 15:30:23 2022
  read: IOPS=440k, BW=215MiB/s (225MB/s)(1504MiB/7001msec)
    slat (nsec): min=1663, max=19526, avg=1853.08, stdev=134.46
    clat (nsec): min=812, max=111990, avg=11416.89, stdev=4588.97
     lat (usec): min=5, max=113, avg=13.27, stdev= 4.58
    clat percentiles (nsec):
     |  1.00th=[ 4704],  5.00th=[ 4896], 10.00th=[ 6624], 20.00th=[ 6816],
     | 30.00th=[ 8768], 40.00th=[ 9152], 50.00th=[11584], 60.00th=[12352],
     | 70.00th=[13888], 80.00th=[15552], 90.00th=[15936], 95.00th=[16512],
     | 99.00th=[26752], 99.50th=[29312], 99.90th=[44288], 99.95th=[60160],
     | 99.99th=[73216]
   bw (  KiB/s): min=219083, max=221465, per=100.00%, avg=219996.93, stdev=649.32, samples=14
   iops        : min=438166, max=442931, avg=439993.93, stdev=1298.76, samples=14
  lat (nsec)   : 1000=0.01%
  lat (usec)   : 4=0.01%, 10=42.30%, 20=55.11%, 50=2.50%, 100=0.08%
  lat (usec)   : 250=0.01%
  cpu          : usr=37.19%, sys=62.80%, ctx=5, majf=0, minf=0
  IO depths    : 1=0.1%, 2=6.7%, 4=73.7%, 8=19.6%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=85.0%, 8=15.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=3079836,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
   READ: bw=215MiB/s (225MB/s), 215MiB/s-215MiB/s (225MB/s-225MB/s), io=1504MiB (1577MB), run=7001-7001msec

$ rm /tmp/fio-output_512_8_libaio-bdev_xnvme_2.txt
rm: cannot remove '/tmp/fio-output_512_8_libaio-bdev_xnvme_2.txt': No such file or directory

$ /opt/fio/bin/fio --name="libaio-bdev_xnvme" --rw="randread" --size="12G" --iodepth="8" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_8_libaio-bdev_xnvme_2.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_512_8_libaio-bdev_xnvme_2.txt
libaio-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666791039,
  "timestamp_ms" : 1666791039010,
  "time" : "Wed Oct 26 15:30:39 2022",
  "jobs" : [
    {
      "jobname" : "libaio-bdev_xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-bdev_xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "8",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_xnvme_libaio.conf",
        "filename" : "Nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 1597813760,
        "io_kbytes" : 1560365,
        "bw_bytes" : 228226504,
        "bw" : 222877,
        "iops" : 445754.320811,
        "runtime" : 7001,
        "total_ios" : 3120726,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1653,
          "max" : 42329,
          "mean" : 1834.530434,
          "stddev" : 111.510378,
          "N" : 3120727
        },
        "clat_ns" : {
          "min" : 791,
          "max" : 125785,
          "mean" : 11024.351078,
          "stddev" : 4765.287623,
          "N" : 3120730,
          "percentile" : {
            "1.000000" : 4640,
            "5.000000" : 4704,
            "10.000000" : 4896,
            "20.000000" : 6688,
            "30.000000" : 8640,
            "40.000000" : 8768,
            "50.000000" : 10688,
            "60.000000" : 11712,
            "70.000000" : 13632,
            "80.000000" : 15424,
            "90.000000" : 15808,
            "95.000000" : 16320,
            "99.000000" : 27520,
            "99.500000" : 29824,
            "99.900000" : 45312,
            "99.950000" : 61184,
            "99.990000" : 74240
          }
        },
        "lat_ns" : {
          "min" : 5721,
          "max" : 127559,
          "mean" : 12858.881482,
          "stddev" : 4762.175625,
          "N" : 3120730
        },
        "bw_min" : 221033,
        "bw_max" : 224499,
        "bw_agg" : 100.000000,
        "bw_mean" : 222992.500000,
        "bw_dev" : 1021.068049,
        "bw_samples" : 14,
        "iops_min" : 442066,
        "iops_max" : 448998,
        "iops_mean" : 445985.000000,
        "iops_stddev" : 2042.136099,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 36.557143,
      "sys_cpu" : 63.414286,
      "ctx" : 13,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.100000,
        "2" : 10.780793,
        "4" : 70.423421,
        "8" : 18.795178,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 85.618620,
        "8" : 14.381380,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.010000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010000,
        "10" : 44.750004,
        "20" : 52.543575,
        "50" : 2.621249,
        "100" : 0.082577,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 8,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

libaio-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=11603: Wed Oct 26 15:30:39 2022
  read: IOPS=446k, BW=218MiB/s (228MB/s)(1524MiB/7001msec)
    slat (nsec): min=1653, max=42329, avg=1834.53, stdev=111.51
    clat (nsec): min=791, max=125785, avg=11024.35, stdev=4765.29
     lat (usec): min=5, max=127, avg=12.86, stdev= 4.76
    clat percentiles (nsec):
     |  1.00th=[ 4640],  5.00th=[ 4704], 10.00th=[ 4896], 20.00th=[ 6688],
     | 30.00th=[ 8640], 40.00th=[ 8768], 50.00th=[10688], 60.00th=[11712],
     | 70.00th=[13632], 80.00th=[15424], 90.00th=[15808], 95.00th=[16320],
     | 99.00th=[27520], 99.50th=[29824], 99.90th=[45312], 99.95th=[61184],
     | 99.99th=[74240]
   bw (  KiB/s): min=221033, max=224499, per=100.00%, avg=222992.50, stdev=1021.07, samples=14
   iops        : min=442066, max=448998, avg=445985.00, stdev=2042.14, samples=14
  lat (nsec)   : 1000=0.01%
  lat (usec)   : 4=0.01%, 10=44.75%, 20=52.54%, 50=2.62%, 100=0.08%
  lat (usec)   : 250=0.01%
  cpu          : usr=36.56%, sys=63.41%, ctx=13, majf=0, minf=0
  IO depths    : 1=0.1%, 2=10.8%, 4=70.4%, 8=18.8%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=85.6%, 8=14.4%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=3120726,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
   READ: bw=218MiB/s (228MB/s), 218MiB/s-218MiB/s (228MB/s-228MB/s), io=1524MiB (1598MB), run=7001-7001msec

$ rm /tmp/fio-output_512_8_libaio-xnvme_0.txt
rm: cannot remove '/tmp/fio-output_512_8_libaio-xnvme_0.txt': No such file or directory

$ /opt/fio/bin/fio --name="libaio-xnvme" --rw="randread" --size="12G" --iodepth="8" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="libaio" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_8_libaio-xnvme_0.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_1_io_uring-bdev_uring_0.txt
io_uring-bdev_uring: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666789319,
  "timestamp_ms" : 1666789319922,
  "time" : "Wed Oct 26 15:01:59 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-bdev_uring",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-bdev_uring",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "1",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_uring_foo.conf",
        "filename" : "Nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 484563456,
        "io_kbytes" : 473206,
        "bw_bytes" : 69213463,
        "bw" : 67591,
        "iops" : 135182.545351,
        "runtime" : 7001,
        "total_ios" : 946413,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 150,
          "max" : 9447,
          "mean" : 172.883611,
          "stddev" : 23.288240,
          "N" : 946413
        },
        "clat_ns" : {
          "min" : 6261,
          "max" : 85470,
          "mean" : 6992.056989,
          "stddev" : 1186.274005,
          "N" : 946413,
          "percentile" : {
            "1.000000" : 6496,
            "5.000000" : 6688,
            "10.000000" : 6752,
            "20.000000" : 6752,
            "30.000000" : 6752,
            "40.000000" : 6816,
            "50.000000" : 6816,
            "60.000000" : 6880,
            "70.000000" : 6880,
            "80.000000" : 6944,
            "90.000000" : 7008,
            "95.000000" : 7136,
            "99.000000" : 13120,
            "99.500000" : 14784,
            "99.900000" : 17024,
            "99.950000" : 17280,
            "99.990000" : 60672
          }
        },
        "lat_ns" : {
          "min" : 6432,
          "max" : 85640,
          "mean" : 7164.940600,
          "stddev" : 1186.749310,
          "N" : 946413
        },
        "bw_min" : 67403,
        "bw_max" : 67839,
        "bw_agg" : 100.000000,
        "bw_mean" : 67627.785714,
        "bw_dev" : 111.226850,
        "bw_samples" : 14,
        "iops_min" : 134806,
        "iops_max" : 135679,
        "iops_mean" : 135255.642857,
        "iops_stddev" : 222.599885,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 53.014286,
      "sys_cpu" : 46.957143,
      "ctx" : 10,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 100.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 98.381151,
        "20" : 1.599090,
        "50" : 0.010000,
        "100" : 0.015955,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 1,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring-bdev_uring: (groupid=0, jobs=1): err= 0: pid=5540: Wed Oct 26 15:01:59 2022
  read: IOPS=135k, BW=66.0MiB/s (69.2MB/s)(462MiB/7001msec)
    slat (nsec): min=150, max=9447, avg=172.88, stdev=23.29
    clat (nsec): min=6261, max=85470, avg=6992.06, stdev=1186.27
     lat (nsec): min=6432, max=85640, avg=7164.94, stdev=1186.75
    clat percentiles (nsec):
     |  1.00th=[ 6496],  5.00th=[ 6688], 10.00th=[ 6752], 20.00th=[ 6752],
     | 30.00th=[ 6752], 40.00th=[ 6816], 50.00th=[ 6816], 60.00th=[ 6880],
     | 70.00th=[ 6880], 80.00th=[ 6944], 90.00th=[ 7008], 95.00th=[ 7136],
     | 99.00th=[13120], 99.50th=[14784], 99.90th=[17024], 99.95th=[17280],
     | 99.99th=[60672]
   bw (  KiB/s): min=67403, max=67839, per=100.00%, avg=67627.79, stdev=111.23, samples=14
   iops        : min=134806, max=135679, avg=135255.64, stdev=222.60, samples=14
  lat (usec)   : 10=98.38%, 20=1.60%, 50=0.01%, 100=0.02%
  cpu          : usr=53.01%, sys=46.96%, ctx=10, majf=0, minf=0
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=946413,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=66.0MiB/s (69.2MB/s), 66.0MiB/s-66.0MiB/s (69.2MB/s-69.2MB/s), io=462MiB (485MB), run=7001-7001msec

$ cat /tmp/fio-output_512_8_libaio-xnvme_0.txt
libaio-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666791053,
  "timestamp_ms" : 1666791053029,
  "time" : "Wed Oct 26 15:30:53 2022",
  "jobs" : [
    {
      "jobname" : "libaio-xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "8",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
        "xnvme_be" : "linux",
        "xnvme_async" : "libaio",
        "xnvme_sync" : "nvme",
        "xnvme_admin" : "nvme",
        "xnvme_dev_nsid" : "1",
        "filename" : "/dev/nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 1495874560,
        "io_kbytes" : 1460815,
        "bw_bytes" : 213665842,
        "bw" : 208658,
        "iops" : 417315.097843,
        "runtime" : 7001,
        "total_ios" : 2921623,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1512,
          "max" : 36950,
          "mean" : 1657.647678,
          "stddev" : 106.702955,
          "N" : 2921623
        },
        "clat_ns" : {
          "min" : 5039,
          "max" : 134552,
          "mean" : 17269.205602,
          "stddev" : 2767.694361,
          "N" : 2921630,
          "percentile" : {
            "1.000000" : 14912,
            "5.000000" : 16512,
            "10.000000" : 16512,
            "20.000000" : 16768,
            "30.000000" : 16768,
            "40.000000" : 16768,
            "50.000000" : 16768,
            "60.000000" : 16768,
            "70.000000" : 16768,
            "80.000000" : 17280,
            "90.000000" : 17536,
            "95.000000" : 18048,
            "99.000000" : 30336,
            "99.500000" : 34560,
            "99.900000" : 50944,
            "99.950000" : 67072,
            "99.990000" : 80384
          }
        },
        "lat_ns" : {
          "min" : 6592,
          "max" : 136265,
          "mean" : 18926.853252,
          "stddev" : 2770.487130,
          "N" : 2921630
        },
        "bw_min" : 208125,
        "bw_max" : 209317,
        "bw_agg" : 100.000000,
        "bw_mean" : 208769.428571,
        "bw_dev" : 418.942389,
        "bw_samples" : 14,
        "iops_min" : 416250,
        "iops_max" : 418634,
        "iops_mean" : 417538.857143,
        "iops_stddev" : 837.884778,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 34.928571,
      "sys_cpu" : 63.642857,
      "ctx" : 16304,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 100.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 99.999983,
        "8" : 0.100000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.213888,
        "20" : 96.208785,
        "50" : 3.475089,
        "100" : 0.101450,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 8,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

libaio-xnvme: (groupid=0, jobs=1): err= 0: pid=11651: Wed Oct 26 15:30:53 2022
  read: IOPS=417k, BW=204MiB/s (214MB/s)(1427MiB/7001msec)
    slat (nsec): min=1512, max=36950, avg=1657.65, stdev=106.70
    clat (usec): min=5, max=134, avg=17.27, stdev= 2.77
     lat (usec): min=6, max=136, avg=18.93, stdev= 2.77
    clat percentiles (nsec):
     |  1.00th=[14912],  5.00th=[16512], 10.00th=[16512], 20.00th=[16768],
     | 30.00th=[16768], 40.00th=[16768], 50.00th=[16768], 60.00th=[16768],
     | 70.00th=[16768], 80.00th=[17280], 90.00th=[17536], 95.00th=[18048],
     | 99.00th=[30336], 99.50th=[34560], 99.90th=[50944], 99.95th=[67072],
     | 99.99th=[80384]
   bw (  KiB/s): min=208125, max=209317, per=100.00%, avg=208769.43, stdev=418.94, samples=14
   iops        : min=416250, max=418634, avg=417538.86, stdev=837.88, samples=14
  lat (usec)   : 10=0.21%, 20=96.21%, 50=3.48%, 100=0.10%, 250=0.01%
  cpu          : usr=34.93%, sys=63.64%, ctx=16304, majf=0, minf=0
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=2921623,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
   READ: bw=204MiB/s (214MB/s), 204MiB/s-204MiB/s (214MB/s-214MB/s), io=1427MiB (1496MB), run=7001-7001msec

$ rm /tmp/fio-output_512_8_libaio-xnvme_1.txt
rm: cannot remove '/tmp/fio-output_512_8_libaio-xnvme_1.txt': No such file or directory

$ /opt/fio/bin/fio --name="libaio-xnvme" --rw="randread" --size="12G" --iodepth="8" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="libaio" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_8_libaio-xnvme_1.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_8_libaio-xnvme_1.txt
libaio-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666791066,
  "timestamp_ms" : 1666791066967,
  "time" : "Wed Oct 26 15:31:06 2022",
  "jobs" : [
    {
      "jobname" : "libaio-xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "8",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
        "xnvme_be" : "linux",
        "xnvme_async" : "libaio",
        "xnvme_sync" : "nvme",
        "xnvme_admin" : "nvme",
        "xnvme_dev_nsid" : "1",
        "filename" : "/dev/nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 1492745216,
        "io_kbytes" : 1457759,
        "bw_bytes" : 213218856,
        "bw" : 208221,
        "iops" : 416442.079703,
        "runtime" : 7001,
        "total_ios" : 2915511,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1502,
          "max" : 33263,
          "mean" : 1671.443877,
          "stddev" : 126.111968,
          "N" : 2915511
        },
        "clat_ns" : {
          "min" : 5090,
          "max" : 134011,
          "mean" : 17299.230179,
          "stddev" : 2704.916413,
          "N" : 2915518,
          "percentile" : {
            "1.000000" : 15040,
            "5.000000" : 16512,
            "10.000000" : 16512,
            "20.000000" : 16768,
            "30.000000" : 16768,
            "40.000000" : 16768,
            "50.000000" : 16768,
            "60.000000" : 16768,
            "70.000000" : 17024,
            "80.000000" : 17280,
            "90.000000" : 17536,
            "95.000000" : 18048,
            "99.000000" : 30336,
            "99.500000" : 34560,
            "99.900000" : 47360,
            "99.950000" : 64256,
            "99.990000" : 79360
          }
        },
        "lat_ns" : {
          "min" : 6653,
          "max" : 135624,
          "mean" : 18970.673988,
          "stddev" : 2707.653440,
          "N" : 2915518
        },
        "bw_min" : 207578,
        "bw_max" : 209884,
        "bw_agg" : 100.000000,
        "bw_mean" : 208259.785714,
        "bw_dev" : 724.737107,
        "bw_samples" : 14,
        "iops_min" : 415156,
        "iops_max" : 419769,
        "iops_mean" : 416519.500000,
        "iops_stddev" : 1449.767660,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 30.614286,
      "sys_cpu" : 67.985714,
      "ctx" : 16266,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 100.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 99.999983,
        "8" : 0.100000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.206173,
        "20" : 96.180395,
        "50" : 3.523842,
        "100" : 0.088767,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 8,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

libaio-xnvme: (groupid=0, jobs=1): err= 0: pid=11700: Wed Oct 26 15:31:06 2022
  read: IOPS=416k, BW=203MiB/s (213MB/s)(1424MiB/7001msec)
    slat (nsec): min=1502, max=33263, avg=1671.44, stdev=126.11
    clat (usec): min=5, max=134, avg=17.30, stdev= 2.70
     lat (usec): min=6, max=135, avg=18.97, stdev= 2.71
    clat percentiles (nsec):
     |  1.00th=[15040],  5.00th=[16512], 10.00th=[16512], 20.00th=[16768],
     | 30.00th=[16768], 40.00th=[16768], 50.00th=[16768], 60.00th=[16768],
     | 70.00th=[17024], 80.00th=[17280], 90.00th=[17536], 95.00th=[18048],
     | 99.00th=[30336], 99.50th=[34560], 99.90th=[47360], 99.95th=[64256],
     | 99.99th=[79360]
   bw (  KiB/s): min=207578, max=209884, per=100.00%, avg=208259.79, stdev=724.74, samples=14
   iops        : min=415156, max=419769, avg=416519.50, stdev=1449.77, samples=14
  lat (usec)   : 10=0.21%, 20=96.18%, 50=3.52%, 100=0.09%, 250=0.01%
  cpu          : usr=30.61%, sys=67.99%, ctx=16266, majf=0, minf=0
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=2915511,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
   READ: bw=203MiB/s (213MB/s), 203MiB/s-203MiB/s (213MB/s-213MB/s), io=1424MiB (1493MB), run=7001-7001msec

$ rm /tmp/fio-output_512_8_libaio-xnvme_2.txt
rm: cannot remove '/tmp/fio-output_512_8_libaio-xnvme_2.txt': No such file or directory

$ /opt/fio/bin/fio --name="libaio-xnvme" --rw="randread" --size="12G" --iodepth="8" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="libaio" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_8_libaio-xnvme_2.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_8_libaio-xnvme_2.txt
libaio-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666791080,
  "timestamp_ms" : 1666791080852,
  "time" : "Wed Oct 26 15:31:20 2022",
  "jobs" : [
    {
      "jobname" : "libaio-xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "8",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
        "xnvme_be" : "linux",
        "xnvme_async" : "libaio",
        "xnvme_sync" : "nvme",
        "xnvme_admin" : "nvme",
        "xnvme_dev_nsid" : "1",
        "filename" : "/dev/nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 1497201664,
        "io_kbytes" : 1462111,
        "bw_bytes" : 213855401,
        "bw" : 208843,
        "iops" : 417685.187830,
        "runtime" : 7001,
        "total_ios" : 2924214,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1513,
          "max" : 25247,
          "mean" : 1654.304958,
          "stddev" : 100.956893,
          "N" : 2924215
        },
        "clat_ns" : {
          "min" : 5080,
          "max" : 137948,
          "mean" : 17253.342018,
          "stddev" : 2675.005836,
          "N" : 2924221,
          "percentile" : {
            "1.000000" : 14784,
            "5.000000" : 16768,
            "10.000000" : 16768,
            "20.000000" : 16768,
            "30.000000" : 16768,
            "40.000000" : 16768,
            "50.000000" : 16768,
            "60.000000" : 16768,
            "70.000000" : 17024,
            "80.000000" : 17024,
            "90.000000" : 17536,
            "95.000000" : 17792,
            "99.000000" : 29568,
            "99.500000" : 33536,
            "99.900000" : 50944,
            "99.950000" : 66048,
            "99.990000" : 79360
          }
        },
        "lat_ns" : {
          "min" : 6703,
          "max" : 139541,
          "mean" : 18907.646970,
          "stddev" : 2674.174544,
          "N" : 2924221
        },
        "bw_min" : 208370,
        "bw_max" : 209183,
        "bw_agg" : 100.000000,
        "bw_mean" : 208880.357143,
        "bw_dev" : 273.334849,
        "bw_samples" : 14,
        "iops_min" : 416741,
        "iops_max" : 418366,
        "iops_mean" : 417760.785714,
        "iops_stddev" : 546.526117,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 32.342857,
      "sys_cpu" : 66.300000,
      "ctx" : 15925,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 100.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 99.999983,
        "8" : 0.100000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.234285,
        "20" : 96.261628,
        "50" : 3.402384,
        "100" : 0.100574,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 8,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

libaio-xnvme: (groupid=0, jobs=1): err= 0: pid=11749: Wed Oct 26 15:31:20 2022
  read: IOPS=418k, BW=204MiB/s (214MB/s)(1428MiB/7001msec)
    slat (nsec): min=1513, max=25247, avg=1654.30, stdev=100.96
    clat (usec): min=5, max=137, avg=17.25, stdev= 2.68
     lat (usec): min=6, max=139, avg=18.91, stdev= 2.67
    clat percentiles (nsec):
     |  1.00th=[14784],  5.00th=[16768], 10.00th=[16768], 20.00th=[16768],
     | 30.00th=[16768], 40.00th=[16768], 50.00th=[16768], 60.00th=[16768],
     | 70.00th=[17024], 80.00th=[17024], 90.00th=[17536], 95.00th=[17792],
     | 99.00th=[29568], 99.50th=[33536], 99.90th=[50944], 99.95th=[66048],
     | 99.99th=[79360]
   bw (  KiB/s): min=208370, max=209183, per=100.00%, avg=208880.36, stdev=273.33, samples=14
   iops        : min=416741, max=418366, avg=417760.79, stdev=546.53, samples=14
  lat (usec)   : 10=0.23%, 20=96.26%, 50=3.40%, 100=0.10%, 250=0.01%
  cpu          : usr=32.34%, sys=66.30%, ctx=15925, majf=0, minf=0
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=2924214,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
   READ: bw=204MiB/s (214MB/s), 204MiB/s-204MiB/s (214MB/s-214MB/s), io=1428MiB (1497MB), run=7001-7001msec

$ rm /tmp/fio-output_512_1_io_uring-bdev_uring_1.txt
rm: cannot remove '/tmp/fio-output_512_1_io_uring-bdev_uring_1.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring-bdev_uring" --rw="randread" --size="12G" --iodepth="1" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_uring_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_1_io_uring-bdev_uring_1.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_512_1_io_uring-bdev_uring_1.txt
io_uring-bdev_uring: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666789335,
  "timestamp_ms" : 1666789335250,
  "time" : "Wed Oct 26 15:02:15 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-bdev_uring",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-bdev_uring",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "1",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_uring_foo.conf",
        "filename" : "Nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 483598848,
        "io_kbytes" : 472264,
        "bw_bytes" : 69075681,
        "bw" : 67456,
        "iops" : 134913.440937,
        "runtime" : 7001,
        "total_ios" : 944529,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 160,
          "max" : 10830,
          "mean" : 176.276544,
          "stddev" : 42.021704,
          "N" : 944529
        },
        "clat_ns" : {
          "min" : 6261,
          "max" : 115436,
          "mean" : 6989.813789,
          "stddev" : 1151.700203,
          "N" : 944529,
          "percentile" : {
            "1.000000" : 6496,
            "5.000000" : 6688,
            "10.000000" : 6752,
            "20.000000" : 6752,
            "30.000000" : 6752,
            "40.000000" : 6816,
            "50.000000" : 6816,
            "60.000000" : 6880,
            "70.000000" : 6880,
            "80.000000" : 6944,
            "90.000000" : 7008,
            "95.000000" : 7136,
            "99.000000" : 13120,
            "99.500000" : 14656,
            "99.900000" : 17024,
            "99.950000" : 17280,
            "99.990000" : 58624
          }
        },
        "lat_ns" : {
          "min" : 6432,
          "max" : 115607,
          "mean" : 7166.090333,
          "stddev" : 1152.795263,
          "N" : 944529
        },
        "bw_min" : 67337,
        "bw_max" : 67822,
        "bw_agg" : 100.000000,
        "bw_mean" : 67490.142857,
        "bw_dev" : 143.619614,
        "bw_samples" : 14,
        "iops_min" : 134674,
        "iops_max" : 135645,
        "iops_mean" : 134980.500000,
        "iops_stddev" : 287.344272,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 54.442857,
      "sys_cpu" : 45.528571,
      "ctx" : 10,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 100.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 98.367546,
        "20" : 1.614985,
        "50" : 0.010000,
        "100" : 0.013128,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 1,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring-bdev_uring: (groupid=0, jobs=1): err= 0: pid=5597: Wed Oct 26 15:02:15 2022
  read: IOPS=135k, BW=65.9MiB/s (69.1MB/s)(461MiB/7001msec)
    slat (nsec): min=160, max=10830, avg=176.28, stdev=42.02
    clat (usec): min=6, max=115, avg= 6.99, stdev= 1.15
     lat (usec): min=6, max=115, avg= 7.17, stdev= 1.15
    clat percentiles (nsec):
     |  1.00th=[ 6496],  5.00th=[ 6688], 10.00th=[ 6752], 20.00th=[ 6752],
     | 30.00th=[ 6752], 40.00th=[ 6816], 50.00th=[ 6816], 60.00th=[ 6880],
     | 70.00th=[ 6880], 80.00th=[ 6944], 90.00th=[ 7008], 95.00th=[ 7136],
     | 99.00th=[13120], 99.50th=[14656], 99.90th=[17024], 99.95th=[17280],
     | 99.99th=[58624]
   bw (  KiB/s): min=67337, max=67822, per=100.00%, avg=67490.14, stdev=143.62, samples=14
   iops        : min=134674, max=135645, avg=134980.50, stdev=287.34, samples=14
  lat (usec)   : 10=98.37%, 20=1.61%, 50=0.01%, 100=0.01%, 250=0.01%
  cpu          : usr=54.44%, sys=45.53%, ctx=10, majf=0, minf=0
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=944529,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=65.9MiB/s (69.1MB/s), 65.9MiB/s-65.9MiB/s (69.1MB/s-69.1MB/s), io=461MiB (484MB), run=7001-7001msec

$ rm /tmp/fio-output_512_1_io_uring-bdev_uring_2.txt
rm: cannot remove '/tmp/fio-output_512_1_io_uring-bdev_uring_2.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring-bdev_uring" --rw="randread" --size="12G" --iodepth="1" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_uring_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_1_io_uring-bdev_uring_2.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_512_1_io_uring-bdev_uring_2.txt
io_uring-bdev_uring: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666789350,
  "timestamp_ms" : 1666789350610,
  "time" : "Wed Oct 26 15:02:30 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-bdev_uring",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-bdev_uring",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "1",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_uring_foo.conf",
        "filename" : "Nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 482136064,
        "io_kbytes" : 470836,
        "bw_bytes" : 68866742,
        "bw" : 67252,
        "iops" : 134505.356378,
        "runtime" : 7001,
        "total_ios" : 941672,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 160,
          "max" : 9859,
          "mean" : 177.184497,
          "stddev" : 34.333593,
          "N" : 941672
        },
        "clat_ns" : {
          "min" : 6301,
          "max" : 114083,
          "mean" : 7024.269096,
          "stddev" : 1168.593145,
          "N" : 941672,
          "percentile" : {
            "1.000000" : 6560,
            "5.000000" : 6752,
            "10.000000" : 6752,
            "20.000000" : 6816,
            "30.000000" : 6816,
            "40.000000" : 6816,
            "50.000000" : 6880,
            "60.000000" : 6880,
            "70.000000" : 6944,
            "80.000000" : 6944,
            "90.000000" : 7072,
            "95.000000" : 7136,
            "99.000000" : 13120,
            "99.500000" : 14656,
            "99.900000" : 17024,
            "99.950000" : 17280,
            "99.990000" : 59136
          }
        },
        "lat_ns" : {
          "min" : 6472,
          "max" : 114264,
          "mean" : 7201.453593,
          "stddev" : 1169.346311,
          "N" : 941672
        },
        "bw_min" : 67168,
        "bw_max" : 67432,
        "bw_agg" : 100.000000,
        "bw_mean" : 67286.928571,
        "bw_dev" : 82.405624,
        "bw_samples" : 14,
        "iops_min" : 134336,
        "iops_max" : 134865,
        "iops_mean" : 134574.000000,
        "iops_stddev" : 164.977854,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 58.614286,
      "sys_cpu" : 41.357143,
      "ctx" : 10,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 100.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 98.349638,
        "20" : 1.631885,
        "50" : 0.010000,
        "100" : 0.014124,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 1,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring-bdev_uring: (groupid=0, jobs=1): err= 0: pid=5654: Wed Oct 26 15:02:30 2022
  read: IOPS=135k, BW=65.7MiB/s (68.9MB/s)(460MiB/7001msec)
    slat (nsec): min=160, max=9859, avg=177.18, stdev=34.33
    clat (usec): min=6, max=114, avg= 7.02, stdev= 1.17
     lat (usec): min=6, max=114, avg= 7.20, stdev= 1.17
    clat percentiles (nsec):
     |  1.00th=[ 6560],  5.00th=[ 6752], 10.00th=[ 6752], 20.00th=[ 6816],
     | 30.00th=[ 6816], 40.00th=[ 6816], 50.00th=[ 6880], 60.00th=[ 6880],
     | 70.00th=[ 6944], 80.00th=[ 6944], 90.00th=[ 7072], 95.00th=[ 7136],
     | 99.00th=[13120], 99.50th=[14656], 99.90th=[17024], 99.95th=[17280],
     | 99.99th=[59136]
   bw (  KiB/s): min=67168, max=67432, per=100.00%, avg=67286.93, stdev=82.41, samples=14
   iops        : min=134336, max=134865, avg=134574.00, stdev=164.98, samples=14
  lat (usec)   : 10=98.35%, 20=1.63%, 50=0.01%, 100=0.01%, 250=0.01%
  cpu          : usr=58.61%, sys=41.36%, ctx=10, majf=0, minf=0
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=941672,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=65.7MiB/s (68.9MB/s), 65.7MiB/s-65.7MiB/s (68.9MB/s-68.9MB/s), io=460MiB (482MB), run=7001-7001msec

$ rm /tmp/fio-output_512_1_io_uring-bdev_xnvme_0.txt
rm: cannot remove '/tmp/fio-output_512_1_io_uring-bdev_xnvme_0.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring-bdev_xnvme" --rw="randread" --size="12G" --iodepth="1" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_1_io_uring-bdev_xnvme_0.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_512_1_io_uring-bdev_xnvme_0.txt
io_uring-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666789366,
  "timestamp_ms" : 1666789366545,
  "time" : "Wed Oct 26 15:02:46 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-bdev_xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-bdev_xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "1",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring.conf",
        "filename" : "Nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 551760384,
        "io_kbytes" : 538828,
        "bw_bytes" : 78811653,
        "bw" : 76964,
        "iops" : 153928.867305,
        "runtime" : 7001,
        "total_ios" : 1077656,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1673,
          "max" : 19667,
          "mean" : 1817.265533,
          "stddev" : 149.648202,
          "N" : 1077657
        },
        "clat_ns" : {
          "min" : 240,
          "max" : 99747,
          "mean" : 4449.052741,
          "stddev" : 1090.885033,
          "N" : 1077656,
          "percentile" : {
            "1.000000" : 4016,
            "5.000000" : 4192,
            "10.000000" : 4256,
            "20.000000" : 4256,
            "30.000000" : 4256,
            "40.000000" : 4320,
            "50.000000" : 4320,
            "60.000000" : 4320,
            "70.000000" : 4384,
            "80.000000" : 4384,
            "90.000000" : 4448,
            "95.000000" : 4512,
            "99.000000" : 9664,
            "99.500000" : 10688,
            "99.900000" : 15296,
            "99.950000" : 15424,
            "99.990000" : 57088
          }
        },
        "lat_ns" : {
          "min" : 5521,
          "max" : 101600,
          "mean" : 6266.318333,
          "stddev" : 1102.612816,
          "N" : 1077656
        },
        "bw_min" : 76849,
        "bw_max" : 77209,
        "bw_agg" : 100.000000,
        "bw_mean" : 77006.785714,
        "bw_dev" : 107.468907,
        "bw_samples" : 14,
        "iops_min" : 153699,
        "iops_max" : 154418,
        "iops_mean" : 154013.714286,
        "iops_stddev" : 214.803104,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 16.271429,
      "sys_cpu" : 83.714286,
      "ctx" : 7,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 100.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.010000,
        "500" : 0.010000,
        "750" : 0.010000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.010000,
        "4" : 0.958284,
        "10" : 98.165184,
        "20" : 0.857880,
        "50" : 0.010000,
        "100" : 0.012249,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 1,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=5712: Wed Oct 26 15:02:46 2022
  read: IOPS=154k, BW=75.2MiB/s (78.8MB/s)(526MiB/7001msec)
    slat (nsec): min=1673, max=19667, avg=1817.27, stdev=149.65
    clat (nsec): min=240, max=99747, avg=4449.05, stdev=1090.89
     lat (usec): min=5, max=101, avg= 6.27, stdev= 1.10
    clat percentiles (nsec):
     |  1.00th=[ 4016],  5.00th=[ 4192], 10.00th=[ 4256], 20.00th=[ 4256],
     | 30.00th=[ 4256], 40.00th=[ 4320], 50.00th=[ 4320], 60.00th=[ 4320],
     | 70.00th=[ 4384], 80.00th=[ 4384], 90.00th=[ 4448], 95.00th=[ 4512],
     | 99.00th=[ 9664], 99.50th=[10688], 99.90th=[15296], 99.95th=[15424],
     | 99.99th=[57088]
   bw (  KiB/s): min=76849, max=77209, per=100.00%, avg=77006.79, stdev=107.47, samples=14
   iops        : min=153699, max=154418, avg=154013.71, stdev=214.80, samples=14
  lat (nsec)   : 250=0.01%, 500=0.01%, 750=0.01%
  lat (usec)   : 2=0.01%, 4=0.96%, 10=98.17%, 20=0.86%, 50=0.01%
  lat (usec)   : 100=0.01%
  cpu          : usr=16.27%, sys=83.71%, ctx=7, majf=0, minf=0
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1077656,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=75.2MiB/s (78.8MB/s), 75.2MiB/s-75.2MiB/s (78.8MB/s-78.8MB/s), io=526MiB (552MB), run=7001-7001msec

$ rm /tmp/fio-output_512_1_io_uring-bdev_xnvme_1.txt
rm: cannot remove '/tmp/fio-output_512_1_io_uring-bdev_xnvme_1.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring-bdev_xnvme" --rw="randread" --size="12G" --iodepth="1" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_1_io_uring-bdev_xnvme_1.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_512_1_io_uring-bdev_xnvme_1.txt
io_uring-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666789382,
  "timestamp_ms" : 1666789382237,
  "time" : "Wed Oct 26 15:03:02 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-bdev_xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-bdev_xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "1",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring.conf",
        "filename" : "Nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 552491520,
        "io_kbytes" : 539542,
        "bw_bytes" : 78916086,
        "bw" : 77066,
        "iops" : 154132.981003,
        "runtime" : 7001,
        "total_ios" : 1079085,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1643,
          "max" : 21780,
          "mean" : 1806.536104,
          "stddev" : 149.212612,
          "N" : 1079085
        },
        "clat_ns" : {
          "min" : 260,
          "max" : 101110,
          "mean" : 4453.978682,
          "stddev" : 1094.494781,
          "N" : 1079085,
          "percentile" : {
            "1.000000" : 4016,
            "5.000000" : 4192,
            "10.000000" : 4256,
            "20.000000" : 4256,
            "30.000000" : 4256,
            "40.000000" : 4320,
            "50.000000" : 4320,
            "60.000000" : 4320,
            "70.000000" : 4384,
            "80.000000" : 4384,
            "90.000000" : 4448,
            "95.000000" : 4576,
            "99.000000" : 9664,
            "99.500000" : 10688,
            "99.900000" : 15296,
            "99.950000" : 15552,
            "99.990000" : 57088
          }
        },
        "lat_ns" : {
          "min" : 5510,
          "max" : 102933,
          "mean" : 6260.514786,
          "stddev" : 1106.115216,
          "N" : 1079085
        },
        "bw_min" : 76941,
        "bw_max" : 77411,
        "bw_agg" : 100.000000,
        "bw_mean" : 77108.928571,
        "bw_dev" : 145.888506,
        "bw_samples" : 14,
        "iops_min" : 153883,
        "iops_max" : 154822,
        "iops_mean" : 154218.000000,
        "iops_stddev" : 291.839524,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 17.014286,
      "sys_cpu" : 82.971429,
      "ctx" : 10,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 100.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.010000,
        "750" : 0.010000,
        "1000" : 0.010000
      },
      "latency_us" : {
        "2" : 0.010000,
        "4" : 0.922634,
        "10" : 98.195230,
        "20" : 0.862768,
        "50" : 0.010000,
        "100" : 0.012140,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 1,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=5770: Wed Oct 26 15:03:02 2022
  read: IOPS=154k, BW=75.3MiB/s (78.9MB/s)(527MiB/7001msec)
    slat (nsec): min=1643, max=21780, avg=1806.54, stdev=149.21
    clat (nsec): min=260, max=101110, avg=4453.98, stdev=1094.49
     lat (usec): min=5, max=102, avg= 6.26, stdev= 1.11
    clat percentiles (nsec):
     |  1.00th=[ 4016],  5.00th=[ 4192], 10.00th=[ 4256], 20.00th=[ 4256],
     | 30.00th=[ 4256], 40.00th=[ 4320], 50.00th=[ 4320], 60.00th=[ 4320],
     | 70.00th=[ 4384], 80.00th=[ 4384], 90.00th=[ 4448], 95.00th=[ 4576],
     | 99.00th=[ 9664], 99.50th=[10688], 99.90th=[15296], 99.95th=[15552],
     | 99.99th=[57088]
   bw (  KiB/s): min=76941, max=77411, per=100.00%, avg=77108.93, stdev=145.89, samples=14
   iops        : min=153883, max=154822, avg=154218.00, stdev=291.84, samples=14
  lat (nsec)   : 500=0.01%, 750=0.01%, 1000=0.01%
  lat (usec)   : 2=0.01%, 4=0.92%, 10=98.20%, 20=0.86%, 50=0.01%
  lat (usec)   : 100=0.01%, 250=0.01%
  cpu          : usr=17.01%, sys=82.97%, ctx=10, majf=0, minf=0
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1079085,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=75.3MiB/s (78.9MB/s), 75.3MiB/s-75.3MiB/s (78.9MB/s-78.9MB/s), io=527MiB (552MB), run=7001-7001msec

$ rm /tmp/fio-output_512_1_io_uring-bdev_xnvme_2.txt
rm: cannot remove '/tmp/fio-output_512_1_io_uring-bdev_xnvme_2.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring-bdev_xnvme" --rw="randread" --size="12G" --iodepth="1" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_1_io_uring-bdev_xnvme_2.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_512_1_io_uring-bdev_xnvme_2.txt
io_uring-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666789397,
  "timestamp_ms" : 1666789397412,
  "time" : "Wed Oct 26 15:03:17 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-bdev_xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-bdev_xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "1",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring.conf",
        "filename" : "Nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 549262848,
        "io_kbytes" : 536389,
        "bw_bytes" : 78454913,
        "bw" : 76616,
        "iops" : 153232.252535,
        "runtime" : 7001,
        "total_ios" : 1072779,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1663,
          "max" : 17694,
          "mean" : 1826.517569,
          "stddev" : 127.585417,
          "N" : 1072779
        },
        "clat_ns" : {
          "min" : 250,
          "max" : 114004,
          "mean" : 4474.261006,
          "stddev" : 1097.446190,
          "N" : 1072779,
          "percentile" : {
            "1.000000" : 4016,
            "5.000000" : 4256,
            "10.000000" : 4256,
            "20.000000" : 4256,
            "30.000000" : 4320,
            "40.000000" : 4320,
            "50.000000" : 4320,
            "60.000000" : 4384,
            "70.000000" : 4384,
            "80.000000" : 4448,
            "90.000000" : 4448,
            "95.000000" : 4576,
            "99.000000" : 9792,
            "99.500000" : 10688,
            "99.900000" : 15296,
            "99.950000" : 15424,
            "99.990000" : 57088
          }
        },
        "lat_ns" : {
          "min" : 5540,
          "max" : 115927,
          "mean" : 6300.778575,
          "stddev" : 1106.768573,
          "N" : 1072779
        },
        "bw_min" : 76414,
        "bw_max" : 76744,
        "bw_agg" : 100.000000,
        "bw_mean" : 76630.071429,
        "bw_dev" : 96.866652,
        "bw_samples" : 14,
        "iops_min" : 152828,
        "iops_max" : 153488,
        "iops_mean" : 153260.357143,
        "iops_stddev" : 193.638683,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 21.700000,
      "sys_cpu" : 78.285714,
      "ctx" : 9,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 100.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.010000,
        "750" : 0.010000,
        "1000" : 0.010000
      },
      "latency_us" : {
        "2" : 0.010000,
        "4" : 0.863552,
        "10" : 98.266745,
        "20" : 0.853577,
        "50" : 0.010000,
        "100" : 0.011466,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 1,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=5827: Wed Oct 26 15:03:17 2022
  read: IOPS=153k, BW=74.8MiB/s (78.5MB/s)(524MiB/7001msec)
    slat (nsec): min=1663, max=17694, avg=1826.52, stdev=127.59
    clat (nsec): min=250, max=114004, avg=4474.26, stdev=1097.45
     lat (usec): min=5, max=115, avg= 6.30, stdev= 1.11
    clat percentiles (nsec):
     |  1.00th=[ 4016],  5.00th=[ 4256], 10.00th=[ 4256], 20.00th=[ 4256],
     | 30.00th=[ 4320], 40.00th=[ 4320], 50.00th=[ 4320], 60.00th=[ 4384],
     | 70.00th=[ 4384], 80.00th=[ 4448], 90.00th=[ 4448], 95.00th=[ 4576],
     | 99.00th=[ 9792], 99.50th=[10688], 99.90th=[15296], 99.95th=[15424],
     | 99.99th=[57088]
   bw (  KiB/s): min=76414, max=76744, per=100.00%, avg=76630.07, stdev=96.87, samples=14
   iops        : min=152828, max=153488, avg=153260.36, stdev=193.64, samples=14
  lat (nsec)   : 500=0.01%, 750=0.01%, 1000=0.01%
  lat (usec)   : 2=0.01%, 4=0.86%, 10=98.27%, 20=0.85%, 50=0.01%
  lat (usec)   : 100=0.01%, 250=0.01%
  cpu          : usr=21.70%, sys=78.29%, ctx=9, majf=0, minf=0
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1072779,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=74.8MiB/s (78.5MB/s), 74.8MiB/s-74.8MiB/s (78.5MB/s-78.5MB/s), io=524MiB (549MB), run=7001-7001msec

$ rm /tmp/fio-output_512_1_io_uring-xnvme_0.txt
rm: cannot remove '/tmp/fio-output_512_1_io_uring-xnvme_0.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring-xnvme" --rw="randread" --size="12G" --iodepth="1" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_1_io_uring-xnvme_0.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_1_io_uring-xnvme_0.txt
io_uring-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666789411,
  "timestamp_ms" : 1666789411269,
  "time" : "Wed Oct 26 15:03:31 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "1",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
        "xnvme_be" : "linux",
        "xnvme_async" : "io_uring",
        "xnvme_sync" : "nvme",
        "xnvme_admin" : "nvme",
        "xnvme_dev_nsid" : "1",
        "filename" : "/dev/nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 390937600,
        "io_kbytes" : 381775,
        "bw_bytes" : 55840251,
        "bw" : 54531,
        "iops" : 109062.991001,
        "runtime" : 7001,
        "total_ios" : 763550,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1503,
          "max" : 14387,
          "mean" : 1667.139119,
          "stddev" : 81.310916,
          "N" : 763550
        },
        "clat_ns" : {
          "min" : 230,
          "max" : 118001,
          "mean" : 7204.880431,
          "stddev" : 960.561063,
          "N" : 763550,
          "percentile" : {
            "1.000000" : 6752,
            "5.000000" : 6944,
            "10.000000" : 6944,
            "20.000000" : 7008,
            "30.000000" : 7008,
            "40.000000" : 7072,
            "50.000000" : 7136,
            "60.000000" : 7136,
            "70.000000" : 7200,
            "80.000000" : 7200,
            "90.000000" : 7264,
            "95.000000" : 7392,
            "99.000000" : 10176,
            "99.500000" : 13504,
            "99.900000" : 15552,
            "99.950000" : 15808,
            "99.990000" : 57600
          }
        },
        "lat_ns" : {
          "min" : 7204,
          "max" : 119654,
          "mean" : 8872.019549,
          "stddev" : 965.822097,
          "N" : 763550
        },
        "bw_min" : 54161,
        "bw_max" : 54881,
        "bw_agg" : 100.000000,
        "bw_mean" : 54540.928571,
        "bw_dev" : 272.108424,
        "bw_samples" : 14,
        "iops_min" : 108322,
        "iops_max" : 109762,
        "iops_mean" : 109081.928571,
        "iops_stddev" : 544.151628,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 8.357143,
      "sys_cpu" : 36.128571,
      "ctx" : 763553,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 100.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010000,
        "10" : 98.860847,
        "20" : 1.121865,
        "50" : 0.010000,
        "100" : 0.013228,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 1,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring-xnvme: (groupid=0, jobs=1): err= 0: pid=5874: Wed Oct 26 15:03:31 2022
  read: IOPS=109k, BW=53.3MiB/s (55.8MB/s)(373MiB/7001msec)
    slat (nsec): min=1503, max=14387, avg=1667.14, stdev=81.31
    clat (nsec): min=230, max=118001, avg=7204.88, stdev=960.56
     lat (nsec): min=7204, max=119654, avg=8872.02, stdev=965.82
    clat percentiles (nsec):
     |  1.00th=[ 6752],  5.00th=[ 6944], 10.00th=[ 6944], 20.00th=[ 7008],
     | 30.00th=[ 7008], 40.00th=[ 7072], 50.00th=[ 7136], 60.00th=[ 7136],
     | 70.00th=[ 7200], 80.00th=[ 7200], 90.00th=[ 7264], 95.00th=[ 7392],
     | 99.00th=[10176], 99.50th=[13504], 99.90th=[15552], 99.95th=[15808],
     | 99.99th=[57600]
   bw (  KiB/s): min=54161, max=54881, per=100.00%, avg=54540.93, stdev=272.11, samples=14
   iops        : min=108322, max=109762, avg=109081.93, stdev=544.15, samples=14
  lat (nsec)   : 250=0.01%
  lat (usec)   : 4=0.01%, 10=98.86%, 20=1.12%, 50=0.01%, 100=0.01%
  lat (usec)   : 250=0.01%
  cpu          : usr=8.36%, sys=36.13%, ctx=763553, majf=0, minf=0
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=763550,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=53.3MiB/s (55.8MB/s), 53.3MiB/s-53.3MiB/s (55.8MB/s-55.8MB/s), io=373MiB (391MB), run=7001-7001msec

$ rm /tmp/fio-output_512_1_io_uring-xnvme_1.txt
rm: cannot remove '/tmp/fio-output_512_1_io_uring-xnvme_1.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring-xnvme" --rw="randread" --size="12G" --iodepth="1" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_1_io_uring-xnvme_1.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_1_io_uring-xnvme_1.txt
io_uring-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666789425,
  "timestamp_ms" : 1666789425191,
  "time" : "Wed Oct 26 15:03:45 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "1",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
        "xnvme_be" : "linux",
        "xnvme_async" : "io_uring",
        "xnvme_sync" : "nvme",
        "xnvme_admin" : "nvme",
        "xnvme_dev_nsid" : "1",
        "filename" : "/dev/nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 390010880,
        "io_kbytes" : 380870,
        "bw_bytes" : 55715840,
        "bw" : 54410,
        "iops" : 108819.857143,
        "runtime" : 7000,
        "total_ios" : 761739,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1523,
          "max" : 23895,
          "mean" : 1676.788178,
          "stddev" : 77.910712,
          "N" : 761740
        },
        "clat_ns" : {
          "min" : 211,
          "max" : 134522,
          "mean" : 7217.694233,
          "stddev" : 1010.392200,
          "N" : 761739,
          "percentile" : {
            "1.000000" : 6752,
            "5.000000" : 6944,
            "10.000000" : 7008,
            "20.000000" : 7008,
            "30.000000" : 7008,
            "40.000000" : 7072,
            "50.000000" : 7136,
            "60.000000" : 7136,
            "70.000000" : 7200,
            "80.000000" : 7200,
            "90.000000" : 7328,
            "95.000000" : 7392,
            "99.000000" : 10176,
            "99.500000" : 13632,
            "99.900000" : 15552,
            "99.950000" : 15808,
            "99.990000" : 59648
          }
        },
        "lat_ns" : {
          "min" : 7003,
          "max" : 136165,
          "mean" : 8894.482495,
          "stddev" : 1014.288753,
          "N" : 761739
        },
        "bw_min" : 54271,
        "bw_max" : 54567,
        "bw_agg" : 100.000000,
        "bw_mean" : 54411.928571,
        "bw_dev" : 78.668207,
        "bw_samples" : 14,
        "iops_min" : 108542,
        "iops_max" : 109134,
        "iops_mean" : 108823.928571,
        "iops_stddev" : 157.274119,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 9.200000,
      "sys_cpu" : 35.271429,
      "ctx" : 761738,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 100.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.010000,
        "500" : 0.010000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010000,
        "10" : 98.872448,
        "20" : 1.107466,
        "50" : 0.010000,
        "100" : 0.015228,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 1,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring-xnvme: (groupid=0, jobs=1): err= 0: pid=5922: Wed Oct 26 15:03:45 2022
  read: IOPS=109k, BW=53.1MiB/s (55.7MB/s)(372MiB/7000msec)
    slat (nsec): min=1523, max=23895, avg=1676.79, stdev=77.91
    clat (nsec): min=211, max=134522, avg=7217.69, stdev=1010.39
     lat (usec): min=7, max=136, avg= 8.89, stdev= 1.01
    clat percentiles (nsec):
     |  1.00th=[ 6752],  5.00th=[ 6944], 10.00th=[ 7008], 20.00th=[ 7008],
     | 30.00th=[ 7008], 40.00th=[ 7072], 50.00th=[ 7136], 60.00th=[ 7136],
     | 70.00th=[ 7200], 80.00th=[ 7200], 90.00th=[ 7328], 95.00th=[ 7392],
     | 99.00th=[10176], 99.50th=[13632], 99.90th=[15552], 99.95th=[15808],
     | 99.99th=[59648]
   bw (  KiB/s): min=54271, max=54567, per=100.00%, avg=54411.93, stdev=78.67, samples=14
   iops        : min=108542, max=109134, avg=108823.93, stdev=157.27, samples=14
  lat (nsec)   : 250=0.01%, 500=0.01%
  lat (usec)   : 4=0.01%, 10=98.87%, 20=1.11%, 50=0.01%, 100=0.02%
  lat (usec)   : 250=0.01%
  cpu          : usr=9.20%, sys=35.27%, ctx=761738, majf=0, minf=0
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=761739,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=53.1MiB/s (55.7MB/s), 53.1MiB/s-53.1MiB/s (55.7MB/s-55.7MB/s), io=372MiB (390MB), run=7000-7000msec

$ rm /tmp/fio-output_512_1_io_uring-xnvme_2.txt
rm: cannot remove '/tmp/fio-output_512_1_io_uring-xnvme_2.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring-xnvme" --rw="randread" --size="12G" --iodepth="1" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_1_io_uring-xnvme_2.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_1_io_uring-xnvme_2.txt
io_uring-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666789439,
  "timestamp_ms" : 1666789439029,
  "time" : "Wed Oct 26 15:03:59 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "1",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
        "xnvme_be" : "linux",
        "xnvme_async" : "io_uring",
        "xnvme_sync" : "nvme",
        "xnvme_admin" : "nvme",
        "xnvme_dev_nsid" : "1",
        "filename" : "/dev/nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 390489088,
        "io_kbytes" : 381337,
        "bw_bytes" : 55776187,
        "bw" : 54468,
        "iops" : 108937.723182,
        "runtime" : 7001,
        "total_ios" : 762673,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1512,
          "max" : 22973,
          "mean" : 1663.808307,
          "stddev" : 81.901888,
          "N" : 762674
        },
        "clat_ns" : {
          "min" : 2965,
          "max" : 339156,
          "mean" : 7211.033603,
          "stddev" : 1048.223194,
          "N" : 762673,
          "percentile" : {
            "1.000000" : 6752,
            "5.000000" : 6944,
            "10.000000" : 6944,
            "20.000000" : 7008,
            "30.000000" : 7008,
            "40.000000" : 7072,
            "50.000000" : 7136,
            "60.000000" : 7136,
            "70.000000" : 7200,
            "80.000000" : 7200,
            "90.000000" : 7264,
            "95.000000" : 7392,
            "99.000000" : 10176,
            "99.500000" : 13504,
            "99.900000" : 15552,
            "99.950000" : 15808,
            "99.990000" : 57600
          }
        },
        "lat_ns" : {
          "min" : 7304,
          "max" : 340799,
          "mean" : 8874.841846,
          "stddev" : 1052.530746,
          "N" : 762673
        },
        "bw_min" : 54216,
        "bw_max" : 54781,
        "bw_agg" : 100.000000,
        "bw_mean" : 54498.571429,
        "bw_dev" : 162.638681,
        "bw_samples" : 14,
        "iops_min" : 108432,
        "iops_max" : 109562,
        "iops_mean" : 108997.285714,
        "iops_stddev" : 325.114283,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 10.714286,
      "sys_cpu" : 33.800000,
      "ctx" : 762675,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 100.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010000,
        "10" : 98.851539,
        "20" : 1.126957,
        "50" : 0.010000,
        "100" : 0.014685,
        "250" : 0.000000,
        "500" : 0.010000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 1,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring-xnvme: (groupid=0, jobs=1): err= 0: pid=5970: Wed Oct 26 15:03:59 2022
  read: IOPS=109k, BW=53.2MiB/s (55.8MB/s)(372MiB/7001msec)
    slat (nsec): min=1512, max=22973, avg=1663.81, stdev=81.90
    clat (usec): min=2, max=339, avg= 7.21, stdev= 1.05
     lat (usec): min=7, max=340, avg= 8.87, stdev= 1.05
    clat percentiles (nsec):
     |  1.00th=[ 6752],  5.00th=[ 6944], 10.00th=[ 6944], 20.00th=[ 7008],
     | 30.00th=[ 7008], 40.00th=[ 7072], 50.00th=[ 7136], 60.00th=[ 7136],
     | 70.00th=[ 7200], 80.00th=[ 7200], 90.00th=[ 7264], 95.00th=[ 7392],
     | 99.00th=[10176], 99.50th=[13504], 99.90th=[15552], 99.95th=[15808],
     | 99.99th=[57600]
   bw (  KiB/s): min=54216, max=54781, per=100.00%, avg=54498.57, stdev=162.64, samples=14
   iops        : min=108432, max=109562, avg=108997.29, stdev=325.11, samples=14
  lat (usec)   : 4=0.01%, 10=98.85%, 20=1.13%, 50=0.01%, 100=0.01%
  lat (usec)   : 500=0.01%
  cpu          : usr=10.71%, sys=33.80%, ctx=762675, majf=0, minf=0
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=762673,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=53.2MiB/s (55.8MB/s), 53.2MiB/s-53.2MiB/s (55.8MB/s-55.8MB/s), io=372MiB (390MB), run=7001-7001msec

$ rm /tmp/fio-output_512_1_libaio-reference_0.txt
rm: cannot remove '/tmp/fio-output_512_1_libaio-reference_0.txt': No such file or directory

$ /opt/fio/bin/fio --name="libaio-reference" --rw="randread" --size="12G" --iodepth="1" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="libaio" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_1_libaio-reference_0.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_1_libaio-reference_0.txt
libaio-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=libaio, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666789452,
  "timestamp_ms" : 1666789452865,
  "time" : "Wed Oct 26 15:04:12 2022",
  "jobs" : [
    {
      "jobname" : "libaio-reference",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-reference",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "1",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "libaio",
        "filename" : "/dev/nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 391555584,
        "io_kbytes" : 382378,
        "bw_bytes" : 55928522,
        "bw" : 54617,
        "iops" : 109235.394944,
        "runtime" : 7001,
        "total_ios" : 764757,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1653,
          "max" : 6341,
          "mean" : 1761.576344,
          "stddev" : 74.909216,
          "N" : 764757
        },
        "clat_ns" : {
          "min" : 2775,
          "max" : 94186,
          "mean" : 7098.314393,
          "stddev" : 981.590607,
          "N" : 764757,
          "percentile" : {
            "1.000000" : 6624,
            "5.000000" : 6816,
            "10.000000" : 6880,
            "20.000000" : 6880,
            "30.000000" : 6944,
            "40.000000" : 6944,
            "50.000000" : 7008,
            "60.000000" : 7072,
            "70.000000" : 7072,
            "80.000000" : 7072,
            "90.000000" : 7200,
            "95.000000" : 7264,
            "99.000000" : 10176,
            "99.500000" : 13504,
            "99.900000" : 15424,
            "99.950000" : 15680,
            "99.990000" : 58112
          }
        },
        "lat_ns" : {
          "min" : 7384,
          "max" : 95980,
          "mean" : 8859.890737,
          "stddev" : 985.744709,
          "N" : 764757
        },
        "bw_min" : 54489,
        "bw_max" : 54838,
        "bw_agg" : 100.000000,
        "bw_mean" : 54647.692308,
        "bw_dev" : 106.501475,
        "bw_samples" : 13,
        "iops_min" : 108978,
        "iops_max" : 109676,
        "iops_mean" : 109295.461538,
        "iops_stddev" : 213.112887,
        "iops_samples" : 13
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 15.628571,
      "sys_cpu" : 27.585714,
      "ctx" : 764759,
      "majf" : 0,
      "minf" : 1,
      "iodepth_level" : {
        "1" : 100.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010000,
        "10" : 98.872583,
        "20" : 1.104534,
        "50" : 0.010000,
        "100" : 0.014253,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 1,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ],
  "disk_util" : [
    {
      "name" : "nvme0n1",
      "read_ios" : 1067722,
      "write_ios" : 0,
      "read_merges" : 0,
      "write_merges" : 0,
      "read_ticks" : 5364,
      "write_ticks" : 0,
      "in_queue" : 5364,
      "util" : 99.045298
    }
  ]
}

libaio-reference: (groupid=0, jobs=1): err= 0: pid=6017: Wed Oct 26 15:04:12 2022
  read: IOPS=109k, BW=53.3MiB/s (55.9MB/s)(373MiB/7001msec)
    slat (nsec): min=1653, max=6341, avg=1761.58, stdev=74.91
    clat (nsec): min=2775, max=94186, avg=7098.31, stdev=981.59
     lat (nsec): min=7384, max=95980, avg=8859.89, stdev=985.74
    clat percentiles (nsec):
     |  1.00th=[ 6624],  5.00th=[ 6816], 10.00th=[ 6880], 20.00th=[ 6880],
     | 30.00th=[ 6944], 40.00th=[ 6944], 50.00th=[ 7008], 60.00th=[ 7072],
     | 70.00th=[ 7072], 80.00th=[ 7072], 90.00th=[ 7200], 95.00th=[ 7264],
     | 99.00th=[10176], 99.50th=[13504], 99.90th=[15424], 99.95th=[15680],
     | 99.99th=[58112]
   bw (  KiB/s): min=54489, max=54838, per=100.00%, avg=54647.69, stdev=106.50, samples=13
   iops        : min=108978, max=109676, avg=109295.46, stdev=213.11, samples=13
  lat (usec)   : 4=0.01%, 10=98.87%, 20=1.10%, 50=0.01%, 100=0.01%
  cpu          : usr=15.63%, sys=27.59%, ctx=764759, majf=0, minf=1
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=764757,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=53.3MiB/s (55.9MB/s), 53.3MiB/s-53.3MiB/s (55.9MB/s-55.9MB/s), io=373MiB (392MB), run=7001-7001msec

Disk stats (read/write):
  nvme0n1: ios=1067722/0, merge=0/0, ticks=5364/0, in_queue=5364, util=99.05%

$ rm /tmp/fio-output_512_1_libaio-reference_1.txt
rm: cannot remove '/tmp/fio-output_512_1_libaio-reference_1.txt': No such file or directory

$ /opt/fio/bin/fio --name="libaio-reference" --rw="randread" --size="12G" --iodepth="1" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="libaio" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_1_libaio-reference_1.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_1_libaio-reference_1.txt
libaio-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=libaio, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666789466,
  "timestamp_ms" : 1666789466728,
  "time" : "Wed Oct 26 15:04:26 2022",
  "jobs" : [
    {
      "jobname" : "libaio-reference",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-reference",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "1",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "libaio",
        "filename" : "/dev/nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 393853440,
        "io_kbytes" : 384622,
        "bw_bytes" : 56256740,
        "bw" : 54938,
        "iops" : 109876.446222,
        "runtime" : 7001,
        "total_ios" : 769245,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1633,
          "max" : 15660,
          "mean" : 1733.714902,
          "stddev" : 59.717360,
          "N" : 769245
        },
        "clat_ns" : {
          "min" : 581,
          "max" : 88846,
          "mean" : 7084.517244,
          "stddev" : 968.322335,
          "N" : 769245,
          "percentile" : {
            "1.000000" : 6624,
            "5.000000" : 6816,
            "10.000000" : 6816,
            "20.000000" : 6880,
            "30.000000" : 6880,
            "40.000000" : 6944,
            "50.000000" : 7008,
            "60.000000" : 7008,
            "70.000000" : 7072,
            "80.000000" : 7072,
            "90.000000" : 7200,
            "95.000000" : 7264,
            "99.000000" : 10176,
            "99.500000" : 13504,
            "99.900000" : 15424,
            "99.950000" : 15680,
            "99.990000" : 58112
          }
        },
        "lat_ns" : {
          "min" : 7294,
          "max" : 90539,
          "mean" : 8818.232146,
          "stddev" : 971.397702,
          "N" : 769245
        },
        "bw_min" : 54727,
        "bw_max" : 55130,
        "bw_agg" : 100.000000,
        "bw_mean" : 54952.357143,
        "bw_dev" : 130.746147,
        "bw_samples" : 14,
        "iops_min" : 109454,
        "iops_max" : 110260,
        "iops_mean" : 109904.714286,
        "iops_stddev" : 261.492294,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 17.442857,
      "sys_cpu" : 25.171429,
      "ctx" : 769247,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 100.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.010000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010000,
        "10" : 98.809872,
        "20" : 1.172708,
        "50" : 0.010000,
        "100" : 0.013650,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 1,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ],
  "disk_util" : [
    {
      "name" : "nvme0n1",
      "read_ios" : 1099355,
      "write_ios" : 0,
      "read_merges" : 0,
      "write_merges" : 0,
      "read_ticks" : 5515,
      "write_ticks" : 0,
      "in_queue" : 5515,
      "util" : 99.069123
    }
  ]
}

libaio-reference: (groupid=0, jobs=1): err= 0: pid=6063: Wed Oct 26 15:04:26 2022
  read: IOPS=110k, BW=53.7MiB/s (56.3MB/s)(376MiB/7001msec)
    slat (nsec): min=1633, max=15660, avg=1733.71, stdev=59.72
    clat (nsec): min=581, max=88846, avg=7084.52, stdev=968.32
     lat (nsec): min=7294, max=90539, avg=8818.23, stdev=971.40
    clat percentiles (nsec):
     |  1.00th=[ 6624],  5.00th=[ 6816], 10.00th=[ 6816], 20.00th=[ 6880],
     | 30.00th=[ 6880], 40.00th=[ 6944], 50.00th=[ 7008], 60.00th=[ 7008],
     | 70.00th=[ 7072], 80.00th=[ 7072], 90.00th=[ 7200], 95.00th=[ 7264],
     | 99.00th=[10176], 99.50th=[13504], 99.90th=[15424], 99.95th=[15680],
     | 99.99th=[58112]
   bw (  KiB/s): min=54727, max=55130, per=100.00%, avg=54952.36, stdev=130.75, samples=14
   iops        : min=109454, max=110260, avg=109904.71, stdev=261.49, samples=14
  lat (nsec)   : 750=0.01%
  lat (usec)   : 4=0.01%, 10=98.81%, 20=1.17%, 50=0.01%, 100=0.01%
  cpu          : usr=17.44%, sys=25.17%, ctx=769247, majf=0, minf=0
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=769245,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=53.7MiB/s (56.3MB/s), 53.7MiB/s-53.7MiB/s (56.3MB/s-56.3MB/s), io=376MiB (394MB), run=7001-7001msec

Disk stats (read/write):
  nvme0n1: ios=1099355/0, merge=0/0, ticks=5515/0, in_queue=5515, util=99.07%

$ rm /tmp/fio-output_512_1_libaio-reference_2.txt
rm: cannot remove '/tmp/fio-output_512_1_libaio-reference_2.txt': No such file or directory

$ /opt/fio/bin/fio --name="libaio-reference" --rw="randread" --size="12G" --iodepth="1" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="libaio" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_1_libaio-reference_2.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_1_libaio-reference_2.txt
libaio-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=libaio, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666789480,
  "timestamp_ms" : 1666789480604,
  "time" : "Wed Oct 26 15:04:40 2022",
  "jobs" : [
    {
      "jobname" : "libaio-reference",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-reference",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "1",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "libaio",
        "filename" : "/dev/nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 392696320,
        "io_kbytes" : 383492,
        "bw_bytes" : 56099474,
        "bw" : 54784,
        "iops" : 109569.285714,
        "runtime" : 7000,
        "total_ios" : 766985,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1613,
          "max" : 9728,
          "mean" : 1717.578988,
          "stddev" : 59.288933,
          "N" : 766985
        },
        "clat_ns" : {
          "min" : 471,
          "max" : 114094,
          "mean" : 7108.687395,
          "stddev" : 1019.143377,
          "N" : 766985,
          "percentile" : {
            "1.000000" : 6624,
            "5.000000" : 6816,
            "10.000000" : 6880,
            "20.000000" : 6880,
            "30.000000" : 6944,
            "40.000000" : 6944,
            "50.000000" : 7008,
            "60.000000" : 7072,
            "70.000000" : 7072,
            "80.000000" : 7136,
            "90.000000" : 7200,
            "95.000000" : 7328,
            "99.000000" : 10176,
            "99.500000" : 13504,
            "99.900000" : 15552,
            "99.950000" : 15680,
            "99.990000" : 60160
          }
        },
        "lat_ns" : {
          "min" : 6893,
          "max" : 115777,
          "mean" : 8826.266383,
          "stddev" : 1021.839338,
          "N" : 766985
        },
        "bw_min" : 54634,
        "bw_max" : 55015,
        "bw_agg" : 100.000000,
        "bw_mean" : 54791.000000,
        "bw_dev" : 131.780704,
        "bw_samples" : 14,
        "iops_min" : 109268,
        "iops_max" : 110030,
        "iops_mean" : 109582.071429,
        "iops_stddev" : 263.670092,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 19.400000,
      "sys_cpu" : 23.571429,
      "ctx" : 766986,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 100.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.010000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010000,
        "10" : 98.838439,
        "20" : 1.141222,
        "50" : 0.010000,
        "100" : 0.016037,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 1,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ],
  "disk_util" : [
    {
      "name" : "nvme0n1",
      "read_ios" : 1099551,
      "write_ios" : 0,
      "read_merges" : 0,
      "write_merges" : 0,
      "read_ticks" : 5504,
      "write_ticks" : 0,
      "in_queue" : 5504,
      "util" : 99.069123
    }
  ]
}

libaio-reference: (groupid=0, jobs=1): err= 0: pid=6110: Wed Oct 26 15:04:40 2022
  read: IOPS=110k, BW=53.5MiB/s (56.1MB/s)(375MiB/7000msec)
    slat (nsec): min=1613, max=9728, avg=1717.58, stdev=59.29
    clat (nsec): min=471, max=114094, avg=7108.69, stdev=1019.14
     lat (usec): min=6, max=115, avg= 8.83, stdev= 1.02
    clat percentiles (nsec):
     |  1.00th=[ 6624],  5.00th=[ 6816], 10.00th=[ 6880], 20.00th=[ 6880],
     | 30.00th=[ 6944], 40.00th=[ 6944], 50.00th=[ 7008], 60.00th=[ 7072],
     | 70.00th=[ 7072], 80.00th=[ 7136], 90.00th=[ 7200], 95.00th=[ 7328],
     | 99.00th=[10176], 99.50th=[13504], 99.90th=[15552], 99.95th=[15680],
     | 99.99th=[60160]
   bw (  KiB/s): min=54634, max=55015, per=100.00%, avg=54791.00, stdev=131.78, samples=14
   iops        : min=109268, max=110030, avg=109582.07, stdev=263.67, samples=14
  lat (nsec)   : 500=0.01%
  lat (usec)   : 4=0.01%, 10=98.84%, 20=1.14%, 50=0.01%, 100=0.02%
  lat (usec)   : 250=0.01%
  cpu          : usr=19.40%, sys=23.57%, ctx=766986, majf=0, minf=0
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=766985,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=53.5MiB/s (56.1MB/s), 53.5MiB/s-53.5MiB/s (56.1MB/s-56.1MB/s), io=375MiB (393MB), run=7000-7000msec

Disk stats (read/write):
  nvme0n1: ios=1099551/0, merge=0/0, ticks=5504/0, in_queue=5504, util=99.07%

$ rm /tmp/fio-output_512_1_libaio-bdev_aio_0.txt
rm: cannot remove '/tmp/fio-output_512_1_libaio-bdev_aio_0.txt': No such file or directory

$ /opt/fio/bin/fio --name="libaio-bdev_aio" --rw="randread" --size="12G" --iodepth="1" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_aio_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_1_libaio-bdev_aio_0.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_512_1_libaio-bdev_aio_0.txt
libaio-bdev_aio: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666789495,
  "timestamp_ms" : 1666789495981,
  "time" : "Wed Oct 26 15:04:55 2022",
  "jobs" : [
    {
      "jobname" : "libaio-bdev_aio",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-bdev_aio",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "1",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_aio_foo.conf",
        "filename" : "Nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 542872576,
        "io_kbytes" : 530149,
        "bw_bytes" : 77553225,
        "bw" : 75735,
        "iops" : 151471.000000,
        "runtime" : 7000,
        "total_ios" : 1060297,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1733,
          "max" : 17563,
          "mean" : 1907.857454,
          "stddev" : 129.896684,
          "N" : 1060298
        },
        "clat_ns" : {
          "min" : 230,
          "max" : 96982,
          "mean" : 4470.611756,
          "stddev" : 1041.750438,
          "N" : 1060297,
          "percentile" : {
            "1.000000" : 4048,
            "5.000000" : 4256,
            "10.000000" : 4256,
            "20.000000" : 4256,
            "30.000000" : 4320,
            "40.000000" : 4320,
            "50.000000" : 4320,
            "60.000000" : 4384,
            "70.000000" : 4384,
            "80.000000" : 4448,
            "90.000000" : 4448,
            "95.000000" : 4640,
            "99.000000" : 9664,
            "99.500000" : 10688,
            "99.900000" : 15296,
            "99.950000" : 15424,
            "99.990000" : 49408
          }
        },
        "lat_ns" : {
          "min" : 5640,
          "max" : 98935,
          "mean" : 6378.469139,
          "stddev" : 1051.660338,
          "N" : 1060297
        },
        "bw_min" : 75451,
        "bw_max" : 76128,
        "bw_agg" : 100.000000,
        "bw_mean" : 75738.142857,
        "bw_dev" : 208.255707,
        "bw_samples" : 14,
        "iops_min" : 150902,
        "iops_max" : 152256,
        "iops_mean" : 151476.285714,
        "iops_stddev" : 416.511414,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 77.700000,
      "sys_cpu" : 22.271429,
      "ctx" : 10,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 100.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.010000,
        "500" : 0.010000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.010000,
        "4" : 0.755637,
        "10" : 98.431383,
        "20" : 0.798361,
        "50" : 0.010000,
        "100" : 0.010000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 1,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

libaio-bdev_aio: (groupid=0, jobs=1): err= 0: pid=6168: Wed Oct 26 15:04:55 2022
  read: IOPS=151k, BW=74.0MiB/s (77.6MB/s)(518MiB/7000msec)
    slat (nsec): min=1733, max=17563, avg=1907.86, stdev=129.90
    clat (nsec): min=230, max=96982, avg=4470.61, stdev=1041.75
     lat (nsec): min=5640, max=98935, avg=6378.47, stdev=1051.66
    clat percentiles (nsec):
     |  1.00th=[ 4048],  5.00th=[ 4256], 10.00th=[ 4256], 20.00th=[ 4256],
     | 30.00th=[ 4320], 40.00th=[ 4320], 50.00th=[ 4320], 60.00th=[ 4384],
     | 70.00th=[ 4384], 80.00th=[ 4448], 90.00th=[ 4448], 95.00th=[ 4640],
     | 99.00th=[ 9664], 99.50th=[10688], 99.90th=[15296], 99.95th=[15424],
     | 99.99th=[49408]
   bw (  KiB/s): min=75451, max=76128, per=100.00%, avg=75738.14, stdev=208.26, samples=14
   iops        : min=150902, max=152256, avg=151476.29, stdev=416.51, samples=14
  lat (nsec)   : 250=0.01%, 500=0.01%
  lat (usec)   : 2=0.01%, 4=0.76%, 10=98.43%, 20=0.80%, 50=0.01%
  lat (usec)   : 100=0.01%
  cpu          : usr=77.70%, sys=22.27%, ctx=10, majf=0, minf=0
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1060297,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=74.0MiB/s (77.6MB/s), 74.0MiB/s-74.0MiB/s (77.6MB/s-77.6MB/s), io=518MiB (543MB), run=7000-7000msec

$ rm /tmp/fio-output_512_1_libaio-bdev_aio_1.txt
rm: cannot remove '/tmp/fio-output_512_1_libaio-bdev_aio_1.txt': No such file or directory

$ /opt/fio/bin/fio --name="libaio-bdev_aio" --rw="randread" --size="12G" --iodepth="1" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_aio_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_1_libaio-bdev_aio_1.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_512_1_libaio-bdev_aio_1.txt
libaio-bdev_aio: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666789511,
  "timestamp_ms" : 1666789511342,
  "time" : "Wed Oct 26 15:05:11 2022",
  "jobs" : [
    {
      "jobname" : "libaio-bdev_aio",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-bdev_aio",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "1",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_aio_foo.conf",
        "filename" : "Nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 544499200,
        "io_kbytes" : 531737,
        "bw_bytes" : 77774489,
        "bw" : 75951,
        "iops" : 151903.299529,
        "runtime" : 7001,
        "total_ios" : 1063475,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1753,
          "max" : 16831,
          "mean" : 1900.398060,
          "stddev" : 139.445188,
          "N" : 1063475
        },
        "clat_ns" : {
          "min" : 210,
          "max" : 77395,
          "mean" : 4459.766281,
          "stddev" : 1077.834294,
          "N" : 1063475,
          "percentile" : {
            "1.000000" : 4048,
            "5.000000" : 4192,
            "10.000000" : 4256,
            "20.000000" : 4256,
            "30.000000" : 4256,
            "40.000000" : 4320,
            "50.000000" : 4320,
            "60.000000" : 4384,
            "70.000000" : 4384,
            "80.000000" : 4384,
            "90.000000" : 4448,
            "95.000000" : 4576,
            "99.000000" : 9664,
            "99.500000" : 10560,
            "99.900000" : 15296,
            "99.950000" : 15424,
            "99.990000" : 57088
          }
        },
        "lat_ns" : {
          "min" : 5620,
          "max" : 79289,
          "mean" : 6360.164341,
          "stddev" : 1089.019246,
          "N" : 1063475
        },
        "bw_min" : 75781,
        "bw_max" : 76168,
        "bw_agg" : 100.000000,
        "bw_mean" : 75991.692308,
        "bw_dev" : 122.716058,
        "bw_samples" : 13,
        "iops_min" : 151562,
        "iops_max" : 152336,
        "iops_mean" : 151983.461538,
        "iops_stddev" : 245.379507,
        "iops_samples" : 13
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 74.900000,
      "sys_cpu" : 25.071429,
      "ctx" : 10,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 100.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.010000,
        "500" : 0.010000,
        "750" : 0.000000,
        "1000" : 0.010000
      },
      "latency_us" : {
        "2" : 0.010000,
        "4" : 0.799643,
        "10" : 98.374292,
        "20" : 0.809046,
        "50" : 0.010000,
        "100" : 0.012130,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 1,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

libaio-bdev_aio: (groupid=0, jobs=1): err= 0: pid=6226: Wed Oct 26 15:05:11 2022
  read: IOPS=152k, BW=74.2MiB/s (77.8MB/s)(519MiB/7001msec)
    slat (nsec): min=1753, max=16831, avg=1900.40, stdev=139.45
    clat (nsec): min=210, max=77395, avg=4459.77, stdev=1077.83
     lat (nsec): min=5620, max=79289, avg=6360.16, stdev=1089.02
    clat percentiles (nsec):
     |  1.00th=[ 4048],  5.00th=[ 4192], 10.00th=[ 4256], 20.00th=[ 4256],
     | 30.00th=[ 4256], 40.00th=[ 4320], 50.00th=[ 4320], 60.00th=[ 4384],
     | 70.00th=[ 4384], 80.00th=[ 4384], 90.00th=[ 4448], 95.00th=[ 4576],
     | 99.00th=[ 9664], 99.50th=[10560], 99.90th=[15296], 99.95th=[15424],
     | 99.99th=[57088]
   bw (  KiB/s): min=75781, max=76168, per=100.00%, avg=75991.69, stdev=122.72, samples=13
   iops        : min=151562, max=152336, avg=151983.46, stdev=245.38, samples=13
  lat (nsec)   : 250=0.01%, 500=0.01%, 1000=0.01%
  lat (usec)   : 2=0.01%, 4=0.80%, 10=98.37%, 20=0.81%, 50=0.01%
  lat (usec)   : 100=0.01%
  cpu          : usr=74.90%, sys=25.07%, ctx=10, majf=0, minf=0
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1063475,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=74.2MiB/s (77.8MB/s), 74.2MiB/s-74.2MiB/s (77.8MB/s-77.8MB/s), io=519MiB (544MB), run=7001-7001msec

$ rm /tmp/fio-output_512_1_libaio-bdev_aio_2.txt
rm: cannot remove '/tmp/fio-output_512_1_libaio-bdev_aio_2.txt': No such file or directory

$ /opt/fio/bin/fio --name="libaio-bdev_aio" --rw="randread" --size="12G" --iodepth="1" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_aio_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_1_libaio-bdev_aio_2.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_512_1_libaio-bdev_aio_2.txt
libaio-bdev_aio: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666789526,
  "timestamp_ms" : 1666789526714,
  "time" : "Wed Oct 26 15:05:26 2022",
  "jobs" : [
    {
      "jobname" : "libaio-bdev_aio",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-bdev_aio",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "1",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_aio_foo.conf",
        "filename" : "Nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 542993920,
        "io_kbytes" : 530267,
        "bw_bytes" : 77559480,
        "bw" : 75741,
        "iops" : 151483.359520,
        "runtime" : 7001,
        "total_ios" : 1060535,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1733,
          "max" : 17993,
          "mean" : 1902.296994,
          "stddev" : 142.855709,
          "N" : 1060535
        },
        "clat_ns" : {
          "min" : 220,
          "max" : 99156,
          "mean" : 4465.624404,
          "stddev" : 1078.405463,
          "N" : 1060535,
          "percentile" : {
            "1.000000" : 4048,
            "5.000000" : 4256,
            "10.000000" : 4256,
            "20.000000" : 4256,
            "30.000000" : 4320,
            "40.000000" : 4320,
            "50.000000" : 4320,
            "60.000000" : 4384,
            "70.000000" : 4384,
            "80.000000" : 4384,
            "90.000000" : 4448,
            "95.000000" : 4576,
            "99.000000" : 9664,
            "99.500000" : 10688,
            "99.900000" : 15296,
            "99.950000" : 15424,
            "99.990000" : 57088
          }
        },
        "lat_ns" : {
          "min" : 5630,
          "max" : 101070,
          "mean" : 6367.921398,
          "stddev" : 1089.091086,
          "N" : 1060535
        },
        "bw_min" : 75607,
        "bw_max" : 75920,
        "bw_agg" : 100.000000,
        "bw_mean" : 75783.928571,
        "bw_dev" : 71.611628,
        "bw_samples" : 14,
        "iops_min" : 151214,
        "iops_max" : 151838,
        "iops_mean" : 151567.857143,
        "iops_stddev" : 142.936703,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 76.042857,
      "sys_cpu" : 23.928571,
      "ctx" : 10,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 100.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.010000,
        "500" : 0.010000,
        "750" : 0.010000,
        "1000" : 0.010000
      },
      "latency_us" : {
        "2" : 0.010000,
        "4" : 0.758202,
        "10" : 98.419666,
        "20" : 0.804028,
        "50" : 0.010000,
        "100" : 0.011881,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 1,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

libaio-bdev_aio: (groupid=0, jobs=1): err= 0: pid=6285: Wed Oct 26 15:05:26 2022
  read: IOPS=151k, BW=74.0MiB/s (77.6MB/s)(518MiB/7001msec)
    slat (nsec): min=1733, max=17993, avg=1902.30, stdev=142.86
    clat (nsec): min=220, max=99156, avg=4465.62, stdev=1078.41
     lat (usec): min=5, max=101, avg= 6.37, stdev= 1.09
    clat percentiles (nsec):
     |  1.00th=[ 4048],  5.00th=[ 4256], 10.00th=[ 4256], 20.00th=[ 4256],
     | 30.00th=[ 4320], 40.00th=[ 4320], 50.00th=[ 4320], 60.00th=[ 4384],
     | 70.00th=[ 4384], 80.00th=[ 4384], 90.00th=[ 4448], 95.00th=[ 4576],
     | 99.00th=[ 9664], 99.50th=[10688], 99.90th=[15296], 99.95th=[15424],
     | 99.99th=[57088]
   bw (  KiB/s): min=75607, max=75920, per=100.00%, avg=75783.93, stdev=71.61, samples=14
   iops        : min=151214, max=151838, avg=151567.86, stdev=142.94, samples=14
  lat (nsec)   : 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
  lat (usec)   : 2=0.01%, 4=0.76%, 10=98.42%, 20=0.80%, 50=0.01%
  lat (usec)   : 100=0.01%
  cpu          : usr=76.04%, sys=23.93%, ctx=10, majf=0, minf=0
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1060535,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=74.0MiB/s (77.6MB/s), 74.0MiB/s-74.0MiB/s (77.6MB/s-77.6MB/s), io=518MiB (543MB), run=7001-7001msec

$ rm /tmp/fio-output_512_1_libaio-bdev_xnvme_0.txt
rm: cannot remove '/tmp/fio-output_512_1_libaio-bdev_xnvme_0.txt': No such file or directory

$ /opt/fio/bin/fio --name="libaio-bdev_xnvme" --rw="randread" --size="12G" --iodepth="1" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_1_libaio-bdev_xnvme_0.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_512_1_libaio-bdev_xnvme_0.txt
libaio-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666789542,
  "timestamp_ms" : 1666789542121,
  "time" : "Wed Oct 26 15:05:42 2022",
  "jobs" : [
    {
      "jobname" : "libaio-bdev_xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-bdev_xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "1",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_xnvme_libaio.conf",
        "filename" : "Nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 523051520,
        "io_kbytes" : 510792,
        "bw_bytes" : 74710972,
        "bw" : 72959,
        "iops" : 145919.868590,
        "runtime" : 7001,
        "total_ios" : 1021585,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1753,
          "max" : 40205,
          "mean" : 1902.474167,
          "stddev" : 112.382836,
          "N" : 1021585
        },
        "clat_ns" : {
          "min" : 591,
          "max" : 101370,
          "mean" : 4716.904850,
          "stddev" : 1098.648880,
          "N" : 1021585,
          "percentile" : {
            "1.000000" : 4384,
            "5.000000" : 4384,
            "10.000000" : 4448,
            "20.000000" : 4448,
            "30.000000" : 4448,
            "40.000000" : 4640,
            "50.000000" : 4640,
            "60.000000" : 4640,
            "70.000000" : 4704,
            "80.000000" : 4704,
            "90.000000" : 4704,
            "95.000000" : 4896,
            "99.000000" : 9664,
            "99.500000" : 10944,
            "99.900000" : 15296,
            "99.950000" : 15424,
            "99.990000" : 58624
          }
        },
        "lat_ns" : {
          "min" : 5871,
          "max" : 103304,
          "mean" : 6619.379017,
          "stddev" : 1103.785829,
          "N" : 1021585
        },
        "bw_min" : 72756,
        "bw_max" : 73193,
        "bw_agg" : 100.000000,
        "bw_mean" : 72972.928571,
        "bw_dev" : 114.125750,
        "bw_samples" : 14,
        "iops_min" : 145512,
        "iops_max" : 146386,
        "iops_mean" : 145945.857143,
        "iops_stddev" : 228.251500,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 54.728571,
      "sys_cpu" : 45.242857,
      "ctx" : 10,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 100.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.010000,
        "1000" : 0.010000
      },
      "latency_us" : {
        "2" : 0.010000,
        "4" : 0.017913,
        "10" : 99.114122,
        "20" : 0.844962,
        "50" : 0.010000,
        "100" : 0.014389,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 1,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

libaio-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=6343: Wed Oct 26 15:05:42 2022
  read: IOPS=146k, BW=71.2MiB/s (74.7MB/s)(499MiB/7001msec)
    slat (nsec): min=1753, max=40205, avg=1902.47, stdev=112.38
    clat (nsec): min=591, max=101370, avg=4716.90, stdev=1098.65
     lat (usec): min=5, max=103, avg= 6.62, stdev= 1.10
    clat percentiles (nsec):
     |  1.00th=[ 4384],  5.00th=[ 4384], 10.00th=[ 4448], 20.00th=[ 4448],
     | 30.00th=[ 4448], 40.00th=[ 4640], 50.00th=[ 4640], 60.00th=[ 4640],
     | 70.00th=[ 4704], 80.00th=[ 4704], 90.00th=[ 4704], 95.00th=[ 4896],
     | 99.00th=[ 9664], 99.50th=[10944], 99.90th=[15296], 99.95th=[15424],
     | 99.99th=[58624]
   bw (  KiB/s): min=72756, max=73193, per=100.00%, avg=72972.93, stdev=114.13, samples=14
   iops        : min=145512, max=146386, avg=145945.86, stdev=228.25, samples=14
  lat (nsec)   : 750=0.01%, 1000=0.01%
  lat (usec)   : 2=0.01%, 4=0.02%, 10=99.11%, 20=0.84%, 50=0.01%
  lat (usec)   : 100=0.01%, 250=0.01%
  cpu          : usr=54.73%, sys=45.24%, ctx=10, majf=0, minf=0
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1021585,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=71.2MiB/s (74.7MB/s), 71.2MiB/s-71.2MiB/s (74.7MB/s-74.7MB/s), io=499MiB (523MB), run=7001-7001msec

$ rm /tmp/fio-output_512_1_libaio-bdev_xnvme_1.txt
rm: cannot remove '/tmp/fio-output_512_1_libaio-bdev_xnvme_1.txt': No such file or directory

$ /opt/fio/bin/fio --name="libaio-bdev_xnvme" --rw="randread" --size="12G" --iodepth="1" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_1_libaio-bdev_xnvme_1.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_512_1_libaio-bdev_xnvme_1.txt
libaio-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666789557,
  "timestamp_ms" : 1666789557225,
  "time" : "Wed Oct 26 15:05:57 2022",
  "jobs" : [
    {
      "jobname" : "libaio-bdev_xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-bdev_xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "1",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_xnvme_libaio.conf",
        "filename" : "Nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 522268672,
        "io_kbytes" : 510028,
        "bw_bytes" : 74609810,
        "bw" : 72861,
        "iops" : 145722.142857,
        "runtime" : 7000,
        "total_ios" : 1020055,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1773,
          "max" : 14247,
          "mean" : 1911.317889,
          "stddev" : 104.278118,
          "N" : 1020056
        },
        "clat_ns" : {
          "min" : 601,
          "max" : 117801,
          "mean" : 4724.615978,
          "stddev" : 1066.997954,
          "N" : 1020055,
          "percentile" : {
            "1.000000" : 4256,
            "5.000000" : 4448,
            "10.000000" : 4448,
            "20.000000" : 4448,
            "30.000000" : 4512,
            "40.000000" : 4512,
            "50.000000" : 4640,
            "60.000000" : 4704,
            "70.000000" : 4704,
            "80.000000" : 4704,
            "90.000000" : 4768,
            "95.000000" : 4768,
            "99.000000" : 9664,
            "99.500000" : 10944,
            "99.900000" : 15296,
            "99.950000" : 15424,
            "99.990000" : 57088
          }
        },
        "lat_ns" : {
          "min" : 5851,
          "max" : 119655,
          "mean" : 6635.933895,
          "stddev" : 1071.288496,
          "N" : 1020055
        },
        "bw_min" : 72757,
        "bw_max" : 73024,
        "bw_agg" : 100.000000,
        "bw_mean" : 72863.785714,
        "bw_dev" : 78.671839,
        "bw_samples" : 14,
        "iops_min" : 145514,
        "iops_max" : 146048,
        "iops_mean" : 145727.642857,
        "iops_stddev" : 157.314779,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 48.328571,
      "sys_cpu" : 51.642857,
      "ctx" : 14,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 100.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.010000,
        "1000" : 0.010000
      },
      "latency_us" : {
        "2" : 0.010000,
        "4" : 0.014607,
        "10" : 99.142791,
        "20" : 0.821034,
        "50" : 0.010000,
        "100" : 0.012646,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 1,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

libaio-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=6400: Wed Oct 26 15:05:57 2022
  read: IOPS=146k, BW=71.2MiB/s (74.6MB/s)(498MiB/7000msec)
    slat (nsec): min=1773, max=14247, avg=1911.32, stdev=104.28
    clat (nsec): min=601, max=117801, avg=4724.62, stdev=1067.00
     lat (usec): min=5, max=119, avg= 6.64, stdev= 1.07
    clat percentiles (nsec):
     |  1.00th=[ 4256],  5.00th=[ 4448], 10.00th=[ 4448], 20.00th=[ 4448],
     | 30.00th=[ 4512], 40.00th=[ 4512], 50.00th=[ 4640], 60.00th=[ 4704],
     | 70.00th=[ 4704], 80.00th=[ 4704], 90.00th=[ 4768], 95.00th=[ 4768],
     | 99.00th=[ 9664], 99.50th=[10944], 99.90th=[15296], 99.95th=[15424],
     | 99.99th=[57088]
   bw (  KiB/s): min=72757, max=73024, per=100.00%, avg=72863.79, stdev=78.67, samples=14
   iops        : min=145514, max=146048, avg=145727.64, stdev=157.31, samples=14
  lat (nsec)   : 750=0.01%, 1000=0.01%
  lat (usec)   : 2=0.01%, 4=0.01%, 10=99.14%, 20=0.82%, 50=0.01%
  lat (usec)   : 100=0.01%, 250=0.01%
  cpu          : usr=48.33%, sys=51.64%, ctx=14, majf=0, minf=0
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1020055,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=71.2MiB/s (74.6MB/s), 71.2MiB/s-71.2MiB/s (74.6MB/s-74.6MB/s), io=498MiB (522MB), run=7000-7000msec

$ rm /tmp/fio-output_512_1_libaio-bdev_xnvme_2.txt
rm: cannot remove '/tmp/fio-output_512_1_libaio-bdev_xnvme_2.txt': No such file or directory

$ /opt/fio/bin/fio --name="libaio-bdev_xnvme" --rw="randread" --size="12G" --iodepth="1" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_1_libaio-bdev_xnvme_2.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_512_1_libaio-bdev_xnvme_2.txt
libaio-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666789572,
  "timestamp_ms" : 1666789572451,
  "time" : "Wed Oct 26 15:06:12 2022",
  "jobs" : [
    {
      "jobname" : "libaio-bdev_xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-bdev_xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "1",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_xnvme_libaio.conf",
        "filename" : "Nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 521175040,
        "io_kbytes" : 508960,
        "bw_bytes" : 74442942,
        "bw" : 72698,
        "iops" : 145396.371947,
        "runtime" : 7001,
        "total_ios" : 1017920,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1743,
          "max" : 14618,
          "mean" : 1896.726667,
          "stddev" : 103.752460,
          "N" : 1017920
        },
        "clat_ns" : {
          "min" : 601,
          "max" : 117551,
          "mean" : 4735.870113,
          "stddev" : 1060.455946,
          "N" : 1017920,
          "percentile" : {
            "1.000000" : 4384,
            "5.000000" : 4384,
            "10.000000" : 4448,
            "20.000000" : 4448,
            "30.000000" : 4640,
            "40.000000" : 4640,
            "50.000000" : 4640,
            "60.000000" : 4640,
            "70.000000" : 4704,
            "80.000000" : 4704,
            "90.000000" : 4704,
            "95.000000" : 4896,
            "99.000000" : 9792,
            "99.500000" : 10816,
            "99.900000" : 15168,
            "99.950000" : 15424,
            "99.990000" : 56576
          }
        },
        "lat_ns" : {
          "min" : 5891,
          "max" : 119544,
          "mean" : 6632.596780,
          "stddev" : 1064.048858,
          "N" : 1017920
        },
        "bw_min" : 72585,
        "bw_max" : 72865,
        "bw_agg" : 100.000000,
        "bw_mean" : 72711.142857,
        "bw_dev" : 64.441813,
        "bw_samples" : 14,
        "iops_min" : 145170,
        "iops_max" : 145730,
        "iops_mean" : 145422.285714,
        "iops_stddev" : 128.883626,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 53.528571,
      "sys_cpu" : 46.457143,
      "ctx" : 10,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 100.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.010000,
        "1000" : 0.010000
      },
      "latency_us" : {
        "2" : 0.010000,
        "4" : 0.016111,
        "10" : 99.099733,
        "20" : 0.863329,
        "50" : 0.010000,
        "100" : 0.011199,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 1,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

libaio-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=6458: Wed Oct 26 15:06:12 2022
  read: IOPS=145k, BW=71.0MiB/s (74.4MB/s)(497MiB/7001msec)
    slat (nsec): min=1743, max=14618, avg=1896.73, stdev=103.75
    clat (nsec): min=601, max=117551, avg=4735.87, stdev=1060.46
     lat (usec): min=5, max=119, avg= 6.63, stdev= 1.06
    clat percentiles (nsec):
     |  1.00th=[ 4384],  5.00th=[ 4384], 10.00th=[ 4448], 20.00th=[ 4448],
     | 30.00th=[ 4640], 40.00th=[ 4640], 50.00th=[ 4640], 60.00th=[ 4640],
     | 70.00th=[ 4704], 80.00th=[ 4704], 90.00th=[ 4704], 95.00th=[ 4896],
     | 99.00th=[ 9792], 99.50th=[10816], 99.90th=[15168], 99.95th=[15424],
     | 99.99th=[56576]
   bw (  KiB/s): min=72585, max=72865, per=100.00%, avg=72711.14, stdev=64.44, samples=14
   iops        : min=145170, max=145730, avg=145422.29, stdev=128.88, samples=14
  lat (nsec)   : 750=0.01%, 1000=0.01%
  lat (usec)   : 2=0.01%, 4=0.02%, 10=99.10%, 20=0.86%, 50=0.01%
  lat (usec)   : 100=0.01%, 250=0.01%
  cpu          : usr=53.53%, sys=46.46%, ctx=10, majf=0, minf=0
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1017920,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=71.0MiB/s (74.4MB/s), 71.0MiB/s-71.0MiB/s (74.4MB/s-74.4MB/s), io=497MiB (521MB), run=7001-7001msec

$ rm /tmp/fio-output_512_1_libaio-xnvme_0.txt
rm: cannot remove '/tmp/fio-output_512_1_libaio-xnvme_0.txt': No such file or directory

$ /opt/fio/bin/fio --name="libaio-xnvme" --rw="randread" --size="12G" --iodepth="1" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="libaio" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_1_libaio-xnvme_0.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_1_libaio-xnvme_0.txt
libaio-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666789586,
  "timestamp_ms" : 1666789586262,
  "time" : "Wed Oct 26 15:06:26 2022",
  "jobs" : [
    {
      "jobname" : "libaio-xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "1",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
        "xnvme_be" : "linux",
        "xnvme_async" : "libaio",
        "xnvme_sync" : "nvme",
        "xnvme_admin" : "nvme",
        "xnvme_dev_nsid" : "1",
        "filename" : "/dev/nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 384857088,
        "io_kbytes" : 375837,
        "bw_bytes" : 54979584,
        "bw" : 53691,
        "iops" : 107381.857143,
        "runtime" : 7000,
        "total_ios" : 751673,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1693,
          "max" : 46567,
          "mean" : 1810.359939,
          "stddev" : 98.872126,
          "N" : 751674
        },
        "clat_ns" : {
          "min" : 612,
          "max" : 75983,
          "mean" : 7221.342058,
          "stddev" : 966.759816,
          "N" : 751673,
          "percentile" : {
            "1.000000" : 6752,
            "5.000000" : 6944,
            "10.000000" : 7008,
            "20.000000" : 7008,
            "30.000000" : 7072,
            "40.000000" : 7072,
            "50.000000" : 7136,
            "60.000000" : 7200,
            "70.000000" : 7200,
            "80.000000" : 7200,
            "90.000000" : 7264,
            "95.000000" : 7392,
            "99.000000" : 10048,
            "99.500000" : 13632,
            "99.900000" : 15424,
            "99.950000" : 15808,
            "99.990000" : 57600
          }
        },
        "lat_ns" : {
          "min" : 7604,
          "max" : 77766,
          "mean" : 9031.702033,
          "stddev" : 973.215408,
          "N" : 751673
        },
        "bw_min" : 53585,
        "bw_max" : 53789,
        "bw_agg" : 100.000000,
        "bw_mean" : 53692.785714,
        "bw_dev" : 71.730455,
        "bw_samples" : 14,
        "iops_min" : 107170,
        "iops_max" : 107578,
        "iops_mean" : 107385.642857,
        "iops_stddev" : 143.549298,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 11.500000,
      "sys_cpu" : 35.457143,
      "ctx" : 751676,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 100.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.010000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010000,
        "10" : 98.988124,
        "20" : 0.992320,
        "50" : 0.010000,
        "100" : 0.014767,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 1,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

libaio-xnvme: (groupid=0, jobs=1): err= 0: pid=6506: Wed Oct 26 15:06:26 2022
  read: IOPS=107k, BW=52.4MiB/s (55.0MB/s)(367MiB/7000msec)
    slat (nsec): min=1693, max=46567, avg=1810.36, stdev=98.87
    clat (nsec): min=612, max=75983, avg=7221.34, stdev=966.76
     lat (nsec): min=7604, max=77766, avg=9031.70, stdev=973.22
    clat percentiles (nsec):
     |  1.00th=[ 6752],  5.00th=[ 6944], 10.00th=[ 7008], 20.00th=[ 7008],
     | 30.00th=[ 7072], 40.00th=[ 7072], 50.00th=[ 7136], 60.00th=[ 7200],
     | 70.00th=[ 7200], 80.00th=[ 7200], 90.00th=[ 7264], 95.00th=[ 7392],
     | 99.00th=[10048], 99.50th=[13632], 99.90th=[15424], 99.95th=[15808],
     | 99.99th=[57600]
   bw (  KiB/s): min=53585, max=53789, per=100.00%, avg=53692.79, stdev=71.73, samples=14
   iops        : min=107170, max=107578, avg=107385.64, stdev=143.55, samples=14
  lat (nsec)   : 750=0.01%
  lat (usec)   : 4=0.01%, 10=98.99%, 20=0.99%, 50=0.01%, 100=0.01%
  cpu          : usr=11.50%, sys=35.46%, ctx=751676, majf=0, minf=0
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=751673,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=52.4MiB/s (55.0MB/s), 52.4MiB/s-52.4MiB/s (55.0MB/s-55.0MB/s), io=367MiB (385MB), run=7000-7000msec

$ rm /tmp/fio-output_512_1_libaio-xnvme_1.txt
rm: cannot remove '/tmp/fio-output_512_1_libaio-xnvme_1.txt': No such file or directory

$ /opt/fio/bin/fio --name="libaio-xnvme" --rw="randread" --size="12G" --iodepth="1" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="libaio" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_1_libaio-xnvme_1.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_1_libaio-xnvme_1.txt
libaio-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666789600,
  "timestamp_ms" : 1666789600159,
  "time" : "Wed Oct 26 15:06:40 2022",
  "jobs" : [
    {
      "jobname" : "libaio-xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "1",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
        "xnvme_be" : "linux",
        "xnvme_async" : "libaio",
        "xnvme_sync" : "nvme",
        "xnvme_admin" : "nvme",
        "xnvme_dev_nsid" : "1",
        "filename" : "/dev/nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 385519104,
        "io_kbytes" : 376483,
        "bw_bytes" : 55066291,
        "bw" : 53775,
        "iops" : 107551.349807,
        "runtime" : 7001,
        "total_ios" : 752967,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1693,
          "max" : 22682,
          "mean" : 1809.645753,
          "stddev" : 88.186978,
          "N" : 752967
        },
        "clat_ns" : {
          "min" : 672,
          "max" : 75631,
          "mean" : 7207.420906,
          "stddev" : 1021.305121,
          "N" : 752967,
          "percentile" : {
            "1.000000" : 6752,
            "5.000000" : 6944,
            "10.000000" : 6944,
            "20.000000" : 7008,
            "30.000000" : 7008,
            "40.000000" : 7072,
            "50.000000" : 7136,
            "60.000000" : 7136,
            "70.000000" : 7200,
            "80.000000" : 7200,
            "90.000000" : 7264,
            "95.000000" : 7392,
            "99.000000" : 10048,
            "99.500000" : 13504,
            "99.900000" : 15424,
            "99.950000" : 15808,
            "99.990000" : 60160
          }
        },
        "lat_ns" : {
          "min" : 7604,
          "max" : 77385,
          "mean" : 9017.066659,
          "stddev" : 1026.795108,
          "N" : 752967
        },
        "bw_min" : 53671,
        "bw_max" : 53904,
        "bw_agg" : 100.000000,
        "bw_mean" : 53804.928571,
        "bw_dev" : 70.010400,
        "bw_samples" : 14,
        "iops_min" : 107342,
        "iops_max" : 107808,
        "iops_mean" : 107610.000000,
        "iops_stddev" : 139.982966,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 11.300000,
      "sys_cpu" : 35.500000,
      "ctx" : 752967,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 100.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.010000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010000,
        "10" : 98.972863,
        "20" : 1.005755,
        "50" : 0.010000,
        "100" : 0.017265,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 1,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

libaio-xnvme: (groupid=0, jobs=1): err= 0: pid=6554: Wed Oct 26 15:06:40 2022
  read: IOPS=108k, BW=52.5MiB/s (55.1MB/s)(368MiB/7001msec)
    slat (nsec): min=1693, max=22682, avg=1809.65, stdev=88.19
    clat (nsec): min=672, max=75631, avg=7207.42, stdev=1021.31
     lat (nsec): min=7604, max=77385, avg=9017.07, stdev=1026.80
    clat percentiles (nsec):
     |  1.00th=[ 6752],  5.00th=[ 6944], 10.00th=[ 6944], 20.00th=[ 7008],
     | 30.00th=[ 7008], 40.00th=[ 7072], 50.00th=[ 7136], 60.00th=[ 7136],
     | 70.00th=[ 7200], 80.00th=[ 7200], 90.00th=[ 7264], 95.00th=[ 7392],
     | 99.00th=[10048], 99.50th=[13504], 99.90th=[15424], 99.95th=[15808],
     | 99.99th=[60160]
   bw (  KiB/s): min=53671, max=53904, per=100.00%, avg=53804.93, stdev=70.01, samples=14
   iops        : min=107342, max=107808, avg=107610.00, stdev=139.98, samples=14
  lat (nsec)   : 750=0.01%
  lat (usec)   : 4=0.01%, 10=98.97%, 20=1.01%, 50=0.01%, 100=0.02%
  cpu          : usr=11.30%, sys=35.50%, ctx=752967, majf=0, minf=0
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=752967,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=52.5MiB/s (55.1MB/s), 52.5MiB/s-52.5MiB/s (55.1MB/s-55.1MB/s), io=368MiB (386MB), run=7001-7001msec

$ rm /tmp/fio-output_512_1_libaio-xnvme_2.txt
rm: cannot remove '/tmp/fio-output_512_1_libaio-xnvme_2.txt': No such file or directory

$ /opt/fio/bin/fio --name="libaio-xnvme" --rw="randread" --size="12G" --iodepth="1" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="libaio" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_1_libaio-xnvme_2.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_1_libaio-xnvme_2.txt
libaio-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666789614,
  "timestamp_ms" : 1666789614006,
  "time" : "Wed Oct 26 15:06:54 2022",
  "jobs" : [
    {
      "jobname" : "libaio-xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "1",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
        "xnvme_be" : "linux",
        "xnvme_async" : "libaio",
        "xnvme_sync" : "nvme",
        "xnvme_admin" : "nvme",
        "xnvme_dev_nsid" : "1",
        "filename" : "/dev/nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 382008832,
        "io_kbytes" : 373055,
        "bw_bytes" : 54564895,
        "bw" : 53286,
        "iops" : 106571.918297,
        "runtime" : 7001,
        "total_ios" : 746110,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1743,
          "max" : 20578,
          "mean" : 1853.316921,
          "stddev" : 84.943754,
          "N" : 746111
        },
        "clat_ns" : {
          "min" : 571,
          "max" : 462377,
          "mean" : 7237.205919,
          "stddev" : 1091.433101,
          "N" : 746110,
          "percentile" : {
            "1.000000" : 6816,
            "5.000000" : 6944,
            "10.000000" : 7008,
            "20.000000" : 7072,
            "30.000000" : 7072,
            "40.000000" : 7072,
            "50.000000" : 7136,
            "60.000000" : 7200,
            "70.000000" : 7200,
            "80.000000" : 7264,
            "90.000000" : 7328,
            "95.000000" : 7456,
            "99.000000" : 9920,
            "99.500000" : 13632,
            "99.900000" : 15424,
            "99.950000" : 16064,
            "99.990000" : 57600
          }
        },
        "lat_ns" : {
          "min" : 6503,
          "max" : 464231,
          "mean" : 9090.522852,
          "stddev" : 1096.371180,
          "N" : 746110
        },
        "bw_min" : 53220,
        "bw_max" : 53390,
        "bw_agg" : 100.000000,
        "bw_mean" : 53313.285714,
        "bw_dev" : 55.447034,
        "bw_samples" : 14,
        "iops_min" : 106440,
        "iops_max" : 106780,
        "iops_mean" : 106626.714286,
        "iops_stddev" : 111.025589,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 13.171429,
      "sys_cpu" : 34.214286,
      "ctx" : 746110,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 100.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.010000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010000,
        "10" : 99.015829,
        "20" : 0.965541,
        "50" : 0.010000,
        "100" : 0.013403,
        "250" : 0.000000,
        "500" : 0.010000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 1,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

libaio-xnvme: (groupid=0, jobs=1): err= 0: pid=6602: Wed Oct 26 15:06:53 2022
  read: IOPS=107k, BW=52.0MiB/s (54.6MB/s)(364MiB/7001msec)
    slat (nsec): min=1743, max=20578, avg=1853.32, stdev=84.94
    clat (nsec): min=571, max=462377, avg=7237.21, stdev=1091.43
     lat (usec): min=6, max=464, avg= 9.09, stdev= 1.10
    clat percentiles (nsec):
     |  1.00th=[ 6816],  5.00th=[ 6944], 10.00th=[ 7008], 20.00th=[ 7072],
     | 30.00th=[ 7072], 40.00th=[ 7072], 50.00th=[ 7136], 60.00th=[ 7200],
     | 70.00th=[ 7200], 80.00th=[ 7264], 90.00th=[ 7328], 95.00th=[ 7456],
     | 99.00th=[ 9920], 99.50th=[13632], 99.90th=[15424], 99.95th=[16064],
     | 99.99th=[57600]
   bw (  KiB/s): min=53220, max=53390, per=100.00%, avg=53313.29, stdev=55.45, samples=14
   iops        : min=106440, max=106780, avg=106626.71, stdev=111.03, samples=14
  lat (nsec)   : 750=0.01%
  lat (usec)   : 4=0.01%, 10=99.02%, 20=0.97%, 50=0.01%, 100=0.01%
  lat (usec)   : 500=0.01%
  cpu          : usr=13.17%, sys=34.21%, ctx=746110, majf=0, minf=0
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=746110,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=52.0MiB/s (54.6MB/s), 52.0MiB/s-52.0MiB/s (54.6MB/s-54.6MB/s), io=364MiB (382MB), run=7001-7001msec

Steps uses worklets. A description of the worklet used by the current step is provided below. The description is extracted from the docstring of the worklet and provided here in verbatim / as-is. 


    Run fio...

{'description': '\n    Run fio...\n',
 'extras': {},
 'id': '004_plot_fio',
 'name': 'plot_fio',
 'nr': 4,
 'status': {'elapsed': 0.7855701446533203,
            'failed': 0,
            'passed': 1,
            'skipped': 0,
            'started': 0.0},
 'uses': 'plot',
 'with': {'path': 'cijoe-output',
          'plot_attributes': 'plot-attributes-512.yaml',
          'tool': 'fio'}}
EOL: In the presence of failure, recall, this too shall pass.